Closed zenn1989 closed 3 years ago
Update: the first run of XSA() going ok, but second and then coming - fails.
Sorry, my bad. It look's like a Rstudio 1.2.xx bug. Run XSA in RGUI - all is ok.
Did you identify what is causing this problem in RStudio? Would be useful to know and see if it can be avoided.
Did you identify what is causing this problem in RStudio? Would be useful to know and see if it can be avoided.
No, i have no idea why it happens, but in Rstudio 1.2.xx always return error after 2nd and coming calls FLXSA. In Rstudio 1.1.xx all works (also in RGUI) ok.
I don't think it's (just) an RStudio problem. I get the same issue when i use R 3.6.1 in RStudio 1.2.5 but it works fine with R 3.5.1 with the same version of RSudio
But this still only happens in Rstudio, right? What do you see if you call for the definition of lapply?
lapply
showMethods("lapply")
Yes, it works fine with r 3.6.1 outside of RStudio, but somehow it does work in Rstudio If you use an older version of R.
showMethods("lapply") Function: lapply (package base) X="ANY" X="FLComps" (inherited from: X="FLlst") X="FLIndices" (inherited from: X="FLlst") X="FLlst" X="FLQuants" (inherited from: X="FLlst") X="list" (inherited from: X="ANY")
From: Iago Mosqueira [mailto:notifications@github.com] Sent: Tuesday 11 February 2020 20:56 To: flr/FLXSA FLXSA@noreply.github.com Cc: Hans Gerritsen hans.Gerritsen@Marine.ie; Comment comment@noreply.github.com Subject: Re: [flr/FLXSA] XSA broken in 2.6.2 release (#6)
But this still only happens in Rstudio, right? What do you see if you call for the definition of lapply?
lapply
showMethods("lapply")
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/flr/FLXSA/issues/6?email_source=notifications&email_token=AHA5GPOBLT4WORMIQZVUXZDRCMGEXA5CNFSM4IOCPJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELOA4UA#issuecomment-584846928, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHA5GPKAATURA53WKKJPTULRCMGEXANCNFSM4IOCPJTQ.
I am afraid I am completely lost here. Not sure it will tell much, but could you call
traceback()
after the error and report the result?
library(FLXSA)
data(ple4) data(ple4.indices) res <- FLXSA(ple4, ple4.indices) res <- FLXSA(ple4, ple4.indices) Error in lapply(assess@index, FLQuant) : negative length vectors are not allowed traceback() 8: lapply(assess@index, FLQuant) 7: initialize(value, ...) 6: initialize(value, ...) 5: new("FLQuants", lapply(assess@index, FLQuant)) 4: fqs(.Call("runFLXSA", stock, indices, control, diag.flag)) 3: .local(stock, indices, ...) 2: FLXSA(ple4, ple4.indices) 1: FLXSA(ple4, ple4.indices)
note that if i then try: lapply(res@index,FLQuant) this works fine Also, i tried 32 bit R in RStudio and that also works fine.
If it's any help: the error does not occur if i do this: res <- FLXSA(ple4, ple4.indices,diag.flag=F)
But when i stepped through the FLXSA function, both of these cause the 'negative length vector' error .Call("runFLXSA", stock, indices, control, diag.flag=T) Error: negative length vectors are not allowed .Call("runFLXSA", stock, indices, control, diag.flag=F) Error: negative length vectors are not allowed
Also getting this error running in the ICES TAF server. Currently using quick fix from @hgerritsen. But this means we do not get any diagnostics. Will send our traceback also.
Hi, the error is coming from the C code:
diag.flag <- TRUE
res <- .Call("runFLXSA", stock, indices, control, diag.flag)
Error: negative length vectors are not allowed
will have a quick look at the C code, but I have not debuged C++ code use with R before...
so it errors when you pass an FLIndices object, but not with an FLIndex
res <- .Call("runFLXSA", xsa.stock, xsa.indices[1], xsa.control, diag.flag)
Error: negative length vectors are not allowed
res <- .Call("runFLXSA", xsa.stock, xsa.indices[[1]], xsa.control, diag.flag)
sorry - last post was not correct:
res <- .Call("runFLXSA", xsa.stock, xsa.indices[[1]], xsa.control, d
iag.flag)
res
[1] "Error in FLIndices"
passing an FLIndex returns an error also, but an expected one.
I have tested on previous versions of FLXSA, but not previous versions of FLCore, or R. But older versions (2 years ago) give same error.
I am testing the version currently in github using
install_github("flr/FLash")
library(FLXSA)
data(ple4)
data(ple4.indices)
res <- FLXSA(ple4, ple4.indices)
and I get no error. If anybody can replicate this I will release a new version.
This is on R3.6.2 and Linux.
Also works on R 3.6.1 in Windows 10
Will give it a go!
I have tried with the two values of diag.flag
res <- FLXSA(ple4, ple4.indices,diag.flag=F)
res <- FLXSA(ple4, ple4.indices,diag.flag=T)
Yep that worked for me too - but it still fails for the objects attached... don't want to go down the wormhole of corupt objects... but have uploaded in case you have time to check with them
download.file("https://github.com/flr/FLXSA/files/4295135/xsa.zip", destfile = "xsa.zip")
# trying URL 'https://github.com/flr/FLXSA/files/4295135/xsa.zip'
# Content type 'application/x-zip-compressed' length 7713 bytes
# downloaded 7713 bytes
unzip("xsa.zip")
library(FLXSA)
# Loading required package: FLCore
# Loading required package: lattice
# Loading required package: iterators
# FLCore (Version 2.6.14, packaged: 2019-11-18 21:54:10 UTC)
# Loading required package: FLAssess
load("xsa.rData")
xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)
# Error in lapply(assess@index, FLQuant) :
# negative length vectors are not allowed
# sessioninfo::session_info()
# - Session info ----------------------------------------------------------
# setting value
# version R version 3.6.3 (2020-02-29)
# os Windows 10 x64
# system x86_64, mingw32
# ui RTerm
# language (EN)
# collate English_United Kingdom.1252
# ctype English_United Kingdom.1252
# tz Europe/Paris
# date 2020-03-05
# - Packages -------------------------------------------------------------------
# ! package * version date lib source
# assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0)
# cli 2.0.2 2020-02-28 [1] CRAN (R 3.6.2)
# crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0)
# fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.2)
# FLAssess * 2.6.3 2019-12-11 [1] local
# FLCore * 2.6.14 2019-11-18 [1] flr (R 3.6.1)
# FLXSA * 2.6.3 2019-11-18 [1] local
# glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.0)
# iterators * 1.0.12 2019-07-26 [1] CRAN (R 3.6.1)
# lattice * 0.20-40 2020-02-19 [1] CRAN (R 3.6.3)
# MASS 7.3-51.5 2019-12-20 [1] CRAN (R 3.6.2)
# R Matrix 1.2-18 <NA> [2] <NA>
# sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.2)
# withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0)
Ha - good luck finding this bug!! error me once, and you wont error me a second time
# R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
# Platform: x86_64-w64-mingw32 (64-bit)
library(FLXSA)
# Loading required package: FLCore
# Loading required package: lattice
# Loading required package: iterators
# FLCore (Version 2.6.14, packaged: 2019-11-18 21:54:10 UTC)
# Loading required package: FLAssess
load("xsa.rData")
xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)
# Error in lapply(assess@index, FLQuant) :
# negative length vectors are not allowed
xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)
# NO ERROR AND RUNS FINE!
you can even reload the data and it will still work. I tried running it up to 7 times and it seems to fail randomly.
So I get it to work using just R. But when I run using radian (python clone of R) and Rstudio it fails.
So another workaround is to do:
res <- try(FLXSA(ple4, ple4.indices, diag = TRUE))
while(inherits(res, "try-error")) {
res <- try(FLXSA(ple4, ple4.indices, diag = TRUE))
}
this atleast gets your diagnostics
Random fails worry me. Smells like a C++ memory allocation thing which is a potential time bomb.
Do we know if it's failing before it reaches the .Call() to the C++ code, or during it?
Has the C++ code ever been updated since Kell first wrote it? I'm not offering, by the way.
Cheers from Auckland!
On Fri, 6 Mar 2020, 13:06 Colin Millar, notifications@github.com wrote:
Ha - good luck finding this bug!! error me once, and you wont error me a second time
R version 3.6.3 (2020-02-29) -- "Holding the Windsock"# Platform: x86_64-w64-mingw32 (64-bit)
library(FLXSA)# Loading required package: FLCore# Loading required package: lattice# Loading required package: iterators# FLCore (Version 2.6.14, packaged: 2019-11-18 21:54:10 UTC)# Loading required package: FLAssess
load("xsa.rData")xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# Error in lapply(assess@index, FLQuant) : # negative length vectors are not allowed xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# NO ERROR AND RUNS FINE!
you can even reload the data and it will still work. I tried running it up to 7 times and it seems to fail randomly.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flr/FLXSA/issues/6?email_source=notifications&email_token=AAKB7FRWHKX6FJLQNF7Z3KLRGA46PA5CNFSM4IOCPJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7BHUQ#issuecomment-595465170, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKB7FTZSECRW7CEUSX6RFLRGA46PANCNFSM4IOCPJTQ .
Looking at Colin's error, it's an R error rather than a C++ crash. Does this occur before the call to C++ or is it when it returns from C++ to R, i.e. some problem with creating the Sexp objects and sending them to R?
On Fri, 6 Mar 2020, 13:23 Finlay Scott, drfinlayscott@gmail.com wrote:
Random fails worry me. Smells like a C++ memory allocation thing which is a potential time bomb.
Do we know if it's failing before it reaches the .Call() to the C++ code, or during it?
Has the C++ code ever been updated since Kell first wrote it? I'm not offering, by the way.
Cheers from Auckland!
On Fri, 6 Mar 2020, 13:06 Colin Millar, notifications@github.com wrote:
Ha - good luck finding this bug!! error me once, and you wont error me a second time
R version 3.6.3 (2020-02-29) -- "Holding the Windsock"# Platform: x86_64-w64-mingw32 (64-bit)
library(FLXSA)# Loading required package: FLCore# Loading required package: lattice# Loading required package: iterators# FLCore (Version 2.6.14, packaged: 2019-11-18 21:54:10 UTC)# Loading required package: FLAssess
load("xsa.rData")xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# Error in lapply(assess@index, FLQuant) : # negative length vectors are not allowed xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# NO ERROR AND RUNS FINE!
you can even reload the data and it will still work. I tried running it up to 7 times and it seems to fail randomly.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flr/FLXSA/issues/6?email_source=notifications&email_token=AAKB7FRWHKX6FJLQNF7Z3KLRGA46PA5CNFSM4IOCPJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7BHUQ#issuecomment-595465170, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKB7FTZSECRW7CEUSX6RFLRGA46PANCNFSM4IOCPJTQ .
Have tried a small test script: https://gist.github.com/colinpmillar/2bc67877dcea0c33e7a087f5aa1e6236
So far finding that if you run .Call("runFLXSA"...
it crashes out every time in rstudio and radian. However, if you run FLXSA(...
it fails more often at the start and seems to get less likely as you go on... though I have not tested this. I was going to set up a bash script to try and explore.
In any case, now that I can relaiably create the error message via:
library(FLXSA)
data(ple4)
data(ple4.indices)
.Call("runFLXSA", ple4, ple4.indices[1], FLXSA.control(), TRUE)
it might help route out the cause
I will have another go. I am also fearful of memory allocation being behind it.
It could be a C++ issue. The diags flag turns off all the diagnostic output when running simulations. There are many arrays in C++ that have to be returned otherwise.
see line 252 onwards in flxsa.R.
Maybe dump assess@index to a file. Then when R crashes have a look at the object.
i.e.
fqs <- function(assess) {
save(a,file="a.RData")
assess@index <- new("FLQuants", lapply(assess@index,FLQuant)) assess@index.hat <- new("FLQuants", lapply(assess@index.hat,FLQuant)) assess@index.var <- new("FLQuants", lapply(assess@index.var,FLQuant)) assess@index.res <- new("FLQuants", lapply(assess@index.res,FLQuant)) assess@q.hat <- new("FLQuants", lapply(assess@q.hat,FLQuant)) assess@q2.hat <- new("FLQuants", lapply(assess@q2.hat,FLQuant))
if (validObject(assess)) return(assess) else stop("not valid") }
Laurie
On 06/03/2020 00:27, Finlay Scott wrote:
Looking at Colin's error, it's an R error rather than a C++ crash. Does this occur before the call to C++ or is it when it returns from C++ to R, i.e. some problem with creating the Sexp objects and sending them to R?
On Fri, 6 Mar 2020, 13:23 Finlay Scott, drfinlayscott@gmail.com wrote:
Random fails worry me. Smells like a C++ memory allocation thing which is a potential time bomb.
Do we know if it's failing before it reaches the .Call() to the C++ code, or during it?
Has the C++ code ever been updated since Kell first wrote it? I'm not offering, by the way.
Cheers from Auckland!
On Fri, 6 Mar 2020, 13:06 Colin Millar, notifications@github.com wrote:
Ha - good luck finding this bug!! error me once, and you wont error me a second time
R version 3.6.3 (2020-02-29) -- "Holding the Windsock"# Platform:
x86_64-w64-mingw32 (64-bit)
library(FLXSA)# Loading required package: FLCore# Loading required package: lattice# Loading required package: iterators# FLCore (Version 2.6.14, packaged: 2019-11-18 21:54:10 UTC)# Loading required package: FLAssess
load("xsa.rData")xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# Error in lapply(assess@index, FLQuant) : # negative length vectors are not allowed xsa <- FLXSA(xsa.stock, xsa.indices, xsa.control, diag = TRUE)# NO ERROR AND RUNS FINE!
you can even reload the data and it will still work. I tried running it up to 7 times and it seems to fail randomly.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub
https://github.com/notifications/unsubscribe-auth/AAKB7FTZSECRW7CEUSX6RFLRGA46PANCNFSM4IOCPJTQ .
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flr/FLXSA/issues/6?email_source=notifications&email_token=AAM2VJ6LDJ5MBJR5FBOLTZDRGA7O7A5CNFSM4IOCPJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7NDGI#issuecomment-595513753, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM2VJ3MRILJM4QPVF6TKB3RGA7O7ANCNFSM4IOCPJTQ.
looks like there is something wrong with object@index[[i]]
What is the value of i and what does object@index[[i]] look like
On 06/03/2020 08:16, Colin Millar wrote:
Something that might be interesting. I don't get the error message when running in R, but when you inspect the element, you get:
res <- .Call("runFLXSA",ple4,ple4.indices[1], FLXSA.control(),TRUE)
res An object of class "FLXSA"
Name: Description: Index.name : Call : new("FLAssess") Quant: age Dims: age year unit season area iter 10 62 1 1 1 1
Range: min max minyear maxyear 0 0 1 1
survivors : [10 62 1 1 1 1 ],units = NA se.int : [9 1 1 1 1 1 ],units = NA se.ext : [9 1 1 1 1 1 ],units = NA n.fshk : [1 1 1 1 1 1 ],units = NA n.nshk : [1 1 1 1 1 1 ],units = NA var.fshk : [1 1 1 1 1 1 ],units = NA var.nshk : [1 1 1 1 1 1 ],units = NA catch.n : [1 1 1 1 1 1 ],units = NA stock.n : [10 61 1 1 1 1 ],units = NA harvest : [10 61 1 1 1 1 ],units = NA : index : [9 11 ] index.res : [9 11 ] index.hat : [9 11 ] index.var : [9 11 ] : Error in object@index[[i]]: attempt to select less than one element in integerOneIndex
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flr/FLXSA/issues/6?email_source=notifications&email_token=AAM2VJ4IVXY2FKNGPFO3A7DRGCWNBA5CNFSM4IOCPJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOAPNKY#issuecomment-595654315, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM2VJ4VWBJMIQG7IYT2VL3RGCWNBANCNFSM4IOCPJTQ.
I still do not get any error when run with ple4
or your dataset when calling FLXSA
res <- FLXSA(xsa.stock,xsa.indices,control=xsa.control)
res <- FLXSA(xsa.stock,xsa.indices,control=xsa.control, TRUE)
If you do
.Call("runFLXSA", xsa.stock, xsa.indices, xsa.control, FALSE)
.Call("runFLXSA", xsa.stock, xsa.indices, xsa.control, TRUE)
then there is an error from summary(FLXSA) because the index FLQuants returned by .Call() does not have names, so summary fails. The names are assigned inside FLXSA() after .Call() returns the result.
But the error I get is
Error in object@index[[i]] :
attempt to select less than one element in integerOneIndex
What happens if you try
res <- .Call("runFLXSA", xsa.stock, xsa.indices, xsa.control, TRUE)
Can you also try openning R from the command line with
R --vanilla
to see if any other package is interfering with [[ for FLlst?
Sorry, just saw your testing code, let me try with that
With FLCore, FLAssess and FLXSA from the FLR repository, I get
> test(ple4, ple4.indices[1:3], control, "FLXSA")
test(ple4, ple4.indices[1:3], control, ".Call")
[1] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
> test(ple4, ple4.indices[1:3], control, ".Call")
[1] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
I get the same result in Linux, R 2.6.3, and Windows 10, R 2.6.1.
Tested in RStudio Version 1.2.5033 in Linux with no problem.
Yeah, me too. Tested in the latest rocker/tidyverse container with rstudio 1.2.5.033
docker run --rm -p 127.0.0.1:8787:8787 -e DISABLE_AUTH=true rocker/tidyverse
with no problems.
So it seems to be windows only, and only when you run FLXSA in rstudio in windows and radian (https://github.com/randy3k/radian) in windows. I suspect that radian in linux will not have the issue, though I don't know.
This is well beyond my knowledge, so my only, pretty lame suggestion is to wrap the .Call("runFLXSA", ...)
in a function
runFLXSA <- function(stock, indices, diag.flag, max.tries = 50) {
res <- try(.Call("runFLXSA", stock, indices, diag.flag))
i <- 0
while (!is.FLXSA(res) && i < max.tries) {
i <- i + 1
res <- try(.Call("runFLXSA", stock, indices, diag.flag))
}
res
}
you could have a Sys.info()[["sysname"]] == "Windows"
in there if you only want to do the try(...)
and while(...) {...}
when running on windows.
tested and working in radian in linux container
docker run -it continuumio/miniconda3 /bin/bash
# https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-language/
conda install r-base
pip install -U radian
radian
now in radian console:
install.packages("iterators", repos = "https://cloud.r-project.org/")
install.packages(c("MASS", "lattice", "Matrix"), repos = "https://cloud.r-project.org/")
install.packages(c("FLCore", "FLAssess", "FLXSA"), repos = "http://flr-project.org/R")
library(FLXSA)
data(ple4)
data(ple4.indices)
control <- FLXSA.control()
res <- .Call("runFLXSA", ple4, ple4.indices[1], control, TRUE)
OK, managed to replicate it, Windows 10, R 3.6.1 RStudio 1.2. And yes, the problem appears to be on the object being returned by .Call() being broken. How could this happen on just this setup I don't know.
The indices object is of length 3, but all 3 FLIndex elements are valid. Running with one of nthem each time still returns the same error.
I have tried @laurieKell suggestion to save or dump the object, but by the time it gets to fqs() it is already invalid and the Rdata cannot be loaded. Any ideas?
Seems that @laurieKell is right about cpp code issue. Error happens only in Windows with Rstudio 1.2.xx (1.1.xx still working great). Bug happens when you call c++ :
obj <- .Call("runFLXSA", stock, indices, control, diag.flag)
and the next call to obj
will throw exception:
negative length vectors are not allowed
any actions with obj
(save or print) will finished with this error.
Look's like we need somebody who got c++ skills to review flxsa.cpp
sources ...
Closing this as it seems to be limited to certain setups. There is little chance of anybody taking this on board. Effort is now placed in other stock assessment models, like FLa4a.
Hello. If you try to perform typical XSA in new version FLXSA package with Rstudio 1.2.xx then you will got error:
You can simply reproduce this bug with test data (ple4):
Is there any solution exist or i should rollback to previous version? Problem occured only in Windows OS and Rstudio 1.2.xx or newst.
UPD: another members in comments found problem source in flxsa.R in fqs() function in lapply() calling from
res <-fqs(.Call("runFLXSA", stock, indices, control, diag.flag))