search

privefl / bigstatsr

R package for statistical tools with big matrices stored on disk.
https://privefl.github.io/bigstatsr/
179 stars 30 forks source link

Error in identical(.self$extptr, NIL_PTR) : object 'NIL_PTR' not found #109

Closed gaochengPRC closed 4 years ago

gaochengPRC commented 4 years ago

Hi!

When I run the program in RStudio, it works so well. But when I run the program via command line (e.g. R CMD BATCH ****.R), the error message is reported as below:

Error in identical(.self$extptr, NIL_PTR) : object 'NIL_PTR' not found Calls: system.time ... extractVec -> $ -> $ -> get -> -> identical Timing stopped at: 8.271 1.844 81.36 Execution halted

Can you tell me why this error occurs and how to address this issue?

Thank you very much!

privefl commented 4 years ago

Please provide a minimal reproducible example.

I tried a script containing 

library(bigstatsr)
G <- big_attachExtdata()
G[, 1]

and I could not reproduce this issue.

privefl commented 4 years ago

I can't run your code if you don't provide data to run the code on. E.g. can you reproduce this error using your code and the data provided in the tutorial?

gaochengPRC commented 4 years ago

My program runs pretty well on the data in the tutorial. However, when it runs on genomic data, it still reports that error. May it be due to the lack of virtual memory? I have no idea about how to handle with this issue.

privefl commented 4 years ago

Could you try to minimize your code to a point where you still get this error and share it with me?

gaochengPRC commented 4 years ago

final_modd <- snp_grid_stacking(multi_PRS_b, y[ind.train], ncores = nb_cores()) # good new_betaa <- final_modd$beta.G # good indd <- which(new_betaa != 0) # good res <- final_modd$intercept + big_prodVec(G.test, new_betaa[indd], ind.col = indd) # wrong Where I get the error has been indicated by bold fonts, say, big_prodVec(....).

Additionally, I get the same error when I try the following:

multi_PRS A Filebacked Big Matrix of type 'float' with 10000 rows and 5600 columns. multi_PRS[] Error in identical(.self$extptr, NIL_PTR) : object 'NIL_PTR' not found multi_PRS[,1] Error in identical(.self$extptr, NIL_PTR) : object 'NIL_PTR' not found

So, I cannot use multi_PRS[a:b, c:d] to select entries out in the Big matrix.

Can you tell me how to handle with this? Thank you very much!

privefl commented 4 years ago

Thanks. But you say that the code in https://privefl.github.io/bigsnpr/articles/SCT.html runs fine?

Was multi_PRS created using an older version of {bigstatsr} maybe?

gaochengPRC commented 4 years ago

Yes. It runs fine. I don't think it's related with the version of bigstatsr. I am now running the program pretty well in RStudio on campus' computer, however, if I run it using command line (e.g., R CMD BATCH ***.R), it will report the error. Also, if I run the program in RStudio on my personal laptop, it will report the same error. So, I feel very confused. Maybe the error is due to the data size !?

privefl commented 4 years ago

I don't understand why you get this error, but I do not think it has to do with memory.

Could you please try to add

NIL_PTR <- bigstatsr:::NIL_PTR

in your code?

gaochengPRC commented 4 years ago

After I add that, it shows Error in get(name, envir = asNamespace(pkg), inherits = FALSE) : object ‘NIL_PTR’ not found

Thanks!

On Sat, Jun 13, 2020 at 1:15 AM Florian Privé notifications@github.com wrote:

I don't understand why you get this error, but I do not think it has to do with memory.

Could you please try to add

NIL_PTR <- bigstatsr:::NIL_PTR

in your code?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/privefl/bigstatsr/issues/109#issuecomment-643571610, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZI22Z2NIWDB4UVYZZPYG3RWMDO5ANCNFSM4M5M2TYQ .

-- ------------------------------------------------------- Wir muessen wissen; wir werden wissen. ----- David Hilbert English Translation: We must know; we will know.

------------------------------------------------------- Department of Mathematical Sciences Michigan Technological University 1400 Townsend Dr, Houghton, MI 49931, U.S.

privefl commented 4 years ago

What packageVersion("bigstatsr") do you have?

gaochengPRC commented 4 years ago

1.0.0

privefl commented 4 years ago

Please install a more recent version.

gaochengPRC commented 4 years ago

Done it. Thank you!

Can you help me with multi_PRS ?

===================

multi_PRS A Filebacked Big Matrix of type 'float' with 10000 rows and 5600 columns.

multi_PRS[1,] Error in getXPtrFBM(path = .self$bk, n = .self$nrow, m = .self$ncol, type = .self$type) : Error when mapping file: No such file or directory.

===================

I saw other users post similar questions to this, I am sure the memory is enough. I remember when I used multi_PRS[1:3,2:5] one month ago, it worked well. I don't know why the error above happened. Do you have any idea about that?

Thank you again, Florian!

On Sat, Jun 13, 2020 at 3:49 PM Florian Privé notifications@github.com wrote:

Please install a more recent version.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/privefl/bigstatsr/issues/109#issuecomment-643669670, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZI224T3VESS67T6FHIY4LRWPJ4XANCNFSM4M5M2TYQ .

-- ------------------------------------------------------- Wir muessen wissen; wir werden wissen. ----- David Hilbert English Translation: We must know; we will know.

------------------------------------------------------- Department of Mathematical Sciences Michigan Technological University 1400 Townsend Dr, Houghton, MI 49931, U.S.

privefl commented 4 years ago

The error is now No such file or directory.. Meaning that the file multi_PRS$bk does not exist. Have you deleted it? Are you using a temporary file?

gaochengPRC commented 4 years ago

multi_PRS$bk [1] "\scores.bk" multi_PRS$rds [1] "\scores.rds"

Both scores.bk and scores.rds are now in the working directory of R. I still have the error, e.g.,

multi_PRS[1:10,1] Error in getXPtrFBM(path = .self$bk, n = .self$nrow, m = .self$ncol, type = .self$type) : Error when mapping file: No such file or directory.

On Sat, Jun 13, 2020 at 6:48 PM Florian Privé notifications@github.com wrote:

The error is now No such file or directory.. Meaning that the file multi_PRS$bk does not exist. Have you deleted it? Are you using a temporary file?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/privefl/bigstatsr/issues/109#issuecomment-643687719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZI22YBMM552J7O6SVSCILRWP633ANCNFSM4M5M2TYQ .

-- ------------------------------------------------------- Wir muessen wissen; wir werden wissen. ----- David Hilbert English Translation: We must know; we will know.

------------------------------------------------------- Department of Mathematical Sciences Michigan Technological University 1400 Townsend Dr, Houghton, MI 49931, U.S.

privefl commented 4 years ago

What do you get for file.exists(multi_PRS$bk)?

gaochengPRC commented 4 years ago

Why is it FALSE?? I have moved scores.bk (multi_PRS$bk) and scores.rds (multi_PRS$rds) to the working directory of R, why does it happen? Thanks!

On Sun, Jun 14, 2020 at 3:19 AM Florian Privé notifications@github.com wrote:

What do you get for file.exists(multi_PRS$bk)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/privefl/bigstatsr/issues/109#issuecomment-643729423, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZI227AA2TITE35QZMXGLDRWR2XJANCNFSM4M5M2TYQ .

-- ------------------------------------------------------- Wir muessen wissen; wir werden wissen. ----- David Hilbert English Translation: We must know; we will know.

------------------------------------------------------- Department of Mathematical Sciences Michigan Technological University 1400 Townsend Dr, Houghton, MI 49931, U.S.

privefl commented 4 years ago

It should be an absolute path. So, I don't quite understand when you say that you moved it to the working directory.

privefl commented 4 years ago

Any update on this?

gaochengPRC commented 4 years ago

No. You can close this request. Thanks!

On Thu, Jul 16, 2020 at 4:13 AM Florian Privé notifications@github.com wrote:

Any update on this?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/privefl/bigstatsr/issues/109#issuecomment-659237523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZI227T6EQ2HEAD6F4FLW3R32ZC3ANCNFSM4M5M2TYQ .

-- ------------------------------------------------------- Wir muessen wissen; wir werden wissen. ----- David Hilbert English Translation: We must know; we will know.

------------------------------------------------------- Department of Mathematical Sciences Michigan Technological University 1400 Townsend Dr, Houghton, MI 49931, U.S.