search

timbitz / Whippet.jl

Lightweight and Fast; RNA-seq quantification at the event-level
MIT License
105 stars 21 forks source link

Distributions.jl failing to precompile #68

Closed seyfim closed 6 years ago

seyfim commented 6 years ago

Hello I am attempting to run whippet-index.jl but keep getting this error message: LoadError: Failed to precompile Distributions to /home/seyfim/.julia/lib/v0.6/Distributions.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:463 [3] require(::Symbol) at ./loading.jl:405 [4] include_from_node1(::String) at ./loading.jl:576 [5] include(::String) at ./sysimg.jl:14 [6] anonymous at ./:2 while loading /home/seyfim/.julia/v0.6/Whippet/src/Whippet.jl, in expression starting on line 17 ERROR: LoadError: Failed to precompile Whippet to /home/seyfim/.julia/lib/v0.6/Whippet.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:497 [3] require(::Symbol) at ./loading.jl:405 [4] include_from_node1(::String) at ./loading.jl:576 [5] include(::String) at ./sysimg.jl:14 [6] process_options(::Base.JLOptions) at ./client.jl:305 [7] _start() at ./client.jl:371 while loading /home/seyfim/.julia/v0.6/Whippet/bin/whippet-index.jl, in expression starting on line 13 I have contacted my HPC admin to see if it was a problem on his end but it was not. Here is my command

!/bin/bash

module load python/3.6.2 module load samtools/current module load julia/0.6.2

julia Pkg.add("Whippet")

julia bin/whippet-index.jl --fasta /home/seyfim/isilon/NGS_Working/Library_Files/Genome/Mouse/mm10/gencode/GRCm38.primary_assembly.genome.fa --gtf /home/seyfim/isilon/NGS_Working/Library_Files/Genome/Mouse/mm10/gencode/gencode.vM16.annotation.gtf --index /home/seyfim/isilon/NGS_Working/Stetson_Thacker/AlternativeSplicing_M3M4/Whippet/data/Index @timbitz

timbitz commented 6 years ago

Hi @seyfim, Happy to try and help!

The output error message (failed to precompile) is fairly uninformative I think but could be from trying to use a package that wasn't installed correctly. I'm wondering if there is/was an earlier message originally output during installation, the first time you ran Pkg.add("Whippet"), or even if you try to uninstall/reinstall Distributions.jl Pkg.rm("Distributions"); Pkg.add("Distributions")?

At first glance the problem appears to be with the installation of a standard dependency "Distributions.jl", however this is curious as they haven't updated to a new version (since 2017) or anything that would be a potential source of friction, and I am unable to reproduce any error on any of my machines (CentOS7.4 & macOS) or Travis CI when installing the current versions just now..

One thought: Can you just nuke the package directory and start over for me: rm -rf /home/seyfim/.julia then open the julia REPL by hand and see if you get errors running Pkg.add("Whippet"). It would also be helpful if you could try this somewhere other than your HPC, like a laptop maybe?

seyfim commented 6 years ago

Hello Dr. Sterne-Weiler,

I tried the following and got the following error:

[seyfim@lri028 bin]$ rm -rf /home/seyfim/.julia/ [seyfim@lri028 bin]$ julia -e 'Pkg.add("Whippet")' INFO: Initializing package repository /home/seyfim/.julia/v0.6 ERROR: SystemError: open: Stale file handle Stacktrace: [1] #systemerror#44 at ./error.jl:64 [inlined] [2] systemerror(::Symbol, ::Bool) at ./error.jl:64 [3] cd(::Base.Pkg.Dir.##8#10{String,String}, ::String) at ./file.jl:67 [4] init(::String, ::String) at ./pkg/dir.jl:53 [5] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at ./pkg/dir.jl:28 [6] add(::String) at ./pkg/pkg.jl:117

Unfortunately, I cannot use a personal laptop for this data, the Cleveland Clinic frowns upon using personal laptops for data analysis and I do not have the permissions to write/install files locally on any of my workstations. I have a linux workstation that I can request to locally install Whippet. Should I try that? Any advice is GREATLY appreciated :)

Thank you so much for all of your help, my colleague and I are exited to see Whippet results!!

Thanks, Marilyn E. Seyfi Bioinformatics Technologist Genomic Medicine Institute Cleveland Clinic-Lerner Research Institute NE5-255 9620 Carnegie Ave Cleveland, OH 44106 work email: seyfim@ccf.orgmailto:seyfim@ccf.org mobile: (216)-633-9418 http://www.lerner.ccf.org/gmi/people/

On May 18, 2018, at 12:29 PM, Tim Sterne-Weiler notifications@github.com<mailto:notifications@github.com> wrote:

@seyfimhttps://github.com/seyfim, it may also help to get any and all the Pkg.add("Whippet") errors with a completely fresh install: rm -rf /home/seyfim/.julia

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/timbitz/Whippet.jl/issues/68#issuecomment-390261405, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AikLJZJGYCtaqjmsaWnDcWdCqMZ--emKks5tzvb1gaJpZM4UEsIu.

===================================

Please consider the environment before printing this e-mail

Cleveland Clinic is currently ranked as the No. 2 hospital in the country by U.S. News & World Report (2017-2018). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.

timbitz commented 6 years ago

Hi Marilyn, @seyfim

It would seem that the SystemError: open: Stale file handle is an NFS problem on your end (see this forum on the topic, or google "stale file handle").. If you can't get past this error, you should probably contact your HPC admin, the error has nothing to do with julia I think...

That being said, maybe close and reopen all your terminals and try again... can you open the julia REPL manually (type julia), and then try the following from the julia console?:

Pkg.add("Distributions")
using Distributions

My guess is that there might an issue with your Distributions.jl install, so I'd be interested if there are any errors here, which I'm afraid may be related to your environment (since I can't reproduce your error on any of 3 linux machines, or 2 macs of various OS/version). If that install of Distributions.jl does work though, then please go ahead and try to install Whippet:

Pkg.add("Whippet")
using Whippet
seyfim commented 6 years ago

Thank you so much for sending those instructions! I was successful at adding the Distributions package as well as the Whippet package. Unfortunately, I tried all of them and still get this error: ERROR: LoadError: Failed to precompile Whippet to /home/seyfim/.julia/lib/v0.6/Whippet.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:497 [3] require(::Symbol) at ./loading.jl:405 [4] include_from_node1(::String) at ./loading.jl:576 [5] include(::String) at ./sysimg.jl:14 [6] process_options(::Base.JLOptions) at ./client.jl:305 [7] _start() at ./client.jl:371 while loading /home/seyfim/.julia/v0.6/Whippet/bin/whippet-index.jl, in expression starting on line 13

I will contact my system admin and see if he can help!

Thank you SO much for all of your help!!!

Thanks, Marilyn E. Seyfi Bioinformatics Technologist Genomic Medicine Institute Cleveland Clinic-Lerner Research Institute NE5-255 9620 Carnegie Ave Cleveland, OH 44106 work email: seyfim@ccf.orgmailto:seyfim@ccf.org mobile: (216)-633-9418 http://www.lerner.ccf.org/gmi/people/

On May 20, 2018, at 1:37 PM, Tim Sterne-Weiler notifications@github.com<mailto:notifications@github.com> wrote:

Hi Marilyn, @seyfimhttps://github.com/seyfim

It would seem that the SystemError: open: Stale file handle is an NFS problem on your end (see this forum on the topichttps://stackoverflow.com/questions/20105260/what-does-stale-file-handle-in-linux-means, or google "stale file handle").. If you can't get past this error, you should probably contact your HPC admin...

That being said, maybe close and reopen all your terminals and try again... can you open the julia REPL manually (type julia), and then try the following from the julia console?:

Pkg.add("Distributions") using Distributions

My guess is that there might an issue with your Distributions.jl install, so I'd be interested if there are any errors here, which I'm afraid may be related to your environment (since I can't reproduce your error on any of 3 linux machines, or 2 macs of various OS/version). If that install of Distributions.jl does work though, then please go ahead and try to install Whippet:

Pkg.add("Whippet") using Whippet

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/timbitz/Whippet.jl/issues/68#issuecomment-390498451, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AikLJRRI_k8R8O_bCms9laktW0N89bb0ks5t0anPgaJpZM4UEsIu.

===================================

Please consider the environment before printing this e-mail

Cleveland Clinic is currently ranked as the No. 2 hospital in the country by U.S. News & World Report (2017-2018). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.

timbitz commented 6 years ago

Hi Marilyn, @seyfim

I'm a little confused since you are sending me output of whippet-index.jl again, which is still failing on the line using Whippet (please note that the using syntax here precompiles the package/s in question and their dependencies). However if I am understanding you correctly, you say that you ran my suggested code, which included the lines using Distributions and using Whippet, and it ran cleanly?

Importantly, after nuking the directory and running Pkg.add("Distributions"), did you run the:

using Distributions

command in julia?

This specifically would let me know whether or not your issue should be moved up to the Distributions package repository (based on the original error message you sent me, it did not appear to be related to whippet-index.jl at all, or even the Whippet code base for that matter.).

That being said, it is very curious that the installations ran cleanly, yet you are having precompilation issues... I wonder if there is something going on with your machine architecture vs. the intended architecture of the julia release compiled you are using?

In the short term I would definitely try that local linux workstation install...

seyfim commented 6 years ago

Okay, so sorry for the confusion!! I definitely did not type using Distributions however, when I do I get this lovely error: julia> using Distributions INFO: Precompiling module Distributions. ERROR: LoadError: Rmath not properly installed. Please run Pkg.build("Rmath") and restart julia Stacktrace: [1] error(::String) at ./error.jl:21 [2] include_from_node1(::String) at ./loading.jl:576 [3] include(::String) at ./sysimg.jl:14 [4] anonymous at ./:2 while loading /home/seyfim/.julia/v0.6/Rmath/src/Rmath.jl, in expression starting on line 17 ERROR: LoadError: LoadError: Failed to precompile Rmath to /home/seyfim/.julia/lib/v0.6/Rmath.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:463 [3] require(::Symbol) at ./loading.jl:405 [4] include_from_node1(::String) at ./loading.jl:576 [5] include(::String) at ./sysimg.jl:14 [6] include_from_node1(::String) at ./loading.jl:576 [7] include(::String) at ./sysimg.jl:14 [8] anonymous at ./:2 while loading /home/seyfim/.julia/v0.6/StatsFuns/src/rmath.jl, in expression starting on line 6 while loading /home/seyfim/.julia/v0.6/StatsFuns/src/StatsFuns.jl, in expression starting on line 228 ERROR: LoadError: Failed to precompile StatsFuns to /home/seyfim/.julia/lib/v0.6/StatsFuns.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:463 [3] require(::Symbol) at ./loading.jl:405 [4] include_from_node1(::String) at ./loading.jl:576 [5] include(::String) at ./sysimg.jl:14 [6] anonymous at ./:2 while loading /home/seyfim/.julia/v0.6/Distributions/src/Distributions.jl, in expression starting on line 6 ERROR: Failed to precompile Distributions to /home/seyfim/.julia/lib/v0.6/Distributions.ji. Stacktrace: [1] compilecache(::String) at ./loading.jl:710 [2] _require(::Symbol) at ./loading.jl:497 [3] require(::Symbol) at ./loading.jl:405

I put in a request for my system admin install Julia on my linux workstation so once it’s installed I will give it a go!

Thanks so much for all of your help and patience, its so rare to find developers willing to help us bioinformaticians! :)

Thanks, Marilyn E. Seyfi Bioinformatics Technologist Genomic Medicine Institute Cleveland Clinic-Lerner Research Institute NE5-255 9620 Carnegie Ave Cleveland, OH 44106 work email: seyfim@ccf.orgmailto:seyfim@ccf.org mobile: (216)-633-9418 http://www.lerner.ccf.org/gmi/people/

On May 21, 2018, at 9:18 AM, Tim Sterne-Weiler notifications@github.com<mailto:notifications@github.com> wrote:

Hi Marilyn, @seyfimhttps://github.com/seyfim

I'm a little confused since you are sending me output of whippet-index.jl again, which is still failing on the line using Whippet (please note that the using syntax here precompiles the package/s in question and their dependencies). However if I am understanding you correctly, you say that you ran my suggested code, which included the lines using Distributions and using Whippet, and it ran cleanly?

Importantly, after nuking the directory and running Pkg.add("Distributions"), did you run the:

using Distributions

command in julia?

This specifically would let me know whether or not your issue should be moved up to the Distributions package repository (based on the original error message you sent me, it did not appear to be related to whippet-index.jl at all, or even the Whippet code base for that matter.).

That being said, it is very curious that the installations ran cleanly, yet you are having precompilation issues... I wonder if there is something going on with your machine architecture vs. the intended architecture of the julia release compiled you are using?

In the short term I would definitely try that local linux workstation install...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/timbitz/Whippet.jl/issues/68#issuecomment-390651200, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AikLJSXOxRP0rqAN1SbwEwqX1DK7BKFUks5t0r6ygaJpZM4UEsIu.

===================================

Please consider the environment before printing this e-mail

Cleveland Clinic is currently ranked as the No. 2 hospital in the country by U.S. News & World Report (2017-2018). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.

seyfim commented 6 years ago

Also just tried to do Pkg.build(“Math”) got this error: julia> Pkg.build("Rmath") INFO: Building Rmath Info: Attempting to create directory /home/seyfim/.julia/v0.6/Rmath/deps/downloads Info: Directory /home/seyfim/.julia/v0.6/Rmath/deps/downloads already exists Info: Downloading file https://github.com/JuliaLang/Rmath-julia/archive/v0.2.0.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (35) SSL connect error ========================================================================[ ERROR: Rmath ]========================================================================

LoadError: failed process: Process(curl -f -o /home/seyfim/.julia/v0.6/Rmath/deps/downloads/v0.2.0.tar.gz -L https://github.com/JuliaLang/Rmath-julia/archive/v0.2.0.tar.gz, ProcessExited(35)) [35] while loading /home/seyfim/.julia/v0.6/Rmath/deps/build.jl, in expression starting on line 47

================================================================================================================================================================

========================================================================[ BUILD ERRORS ]========================================================================

WARNING: Rmath had build errors.

================================================================================================================================================================ Yay bioinformatics! Will keep you posted Thanks, Marilyn E. Seyfi Bioinformatics Technologist Genomic Medicine Institute Cleveland Clinic-Lerner Research Institute NE5-255 9620 Carnegie Ave Cleveland, OH 44106 work email: seyfim@ccf.orgmailto:seyfim@ccf.org mobile: (216)-633-9418 http://www.lerner.ccf.org/gmi/people/

On May 21, 2018, at 9:18 AM, Tim Sterne-Weiler notifications@github.com<mailto:notifications@github.com> wrote:

Hi Marilyn, @seyfimhttps://github.com/seyfim

I'm a little confused since you are sending me output of whippet-index.jl again, which is still failing on the line using Whippet (please note that the using syntax here precompiles the package/s in question and their dependencies). However if I am understanding you correctly, you say that you ran my suggested code, which included the lines using Distributions and using Whippet, and it ran cleanly?

Importantly, after nuking the directory and running Pkg.add("Distributions"), did you run the:

using Distributions

command in julia?

This specifically would let me know whether or not your issue should be moved up to the Distributions package repository (based on the original error message you sent me, it did not appear to be related to whippet-index.jl at all, or even the Whippet code base for that matter.).

That being said, it is very curious that the installations ran cleanly, yet you are having precompilation issues... I wonder if there is something going on with your machine architecture vs. the intended architecture of the julia release compiled you are using?

In the short term I would definitely try that local linux workstation install...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/timbitz/Whippet.jl/issues/68#issuecomment-390651200, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AikLJSXOxRP0rqAN1SbwEwqX1DK7BKFUks5t0r6ygaJpZM4UEsIu.

===================================

Please consider the environment before printing this e-mail

Cleveland Clinic is currently ranked as the No. 2 hospital in the country by U.S. News & World Report (2017-2018). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.

timbitz commented 6 years ago

Hi Marilyn, @seyfim

OK. Yes that is what I thought, your trouble is with Distributions.jl, but specifically its dependency Rmath.jl. To be fair however, the error here seems to actually be just with downloading the Rmath tarball using curl. Since its not reproducible on any machine I have access to, it is probably not likely to be for the developers of those packages either, though they are very likely to have some ideas, so it might be worth opening an issue with just the error from Pkg.build("Rmath") at Rmath.jl.

My guess is that your sysadmin can probably also help... as the error is probably due to versioning requirements and the libcurl libraries (outdated OS?), or a firewall, or some other install like anaconda messing with curl, or other friction.

But I think this definitely isn't specifically related to Whippet in any way, or a typical error, and is very likely specific to a non-standard environment.

seyfim commented 6 years ago

Hey I just wanted to let you know that when I ran locally on my linux workstation I was able to go through the entire pipeline of tools! Thank you so much for all of your help and we are very excited as we found concordance with other tools (rMATs and MAJIQ). Thank you again!!

Thanks, Marilyn E. Seyfi Bioinformatics Technologist Genomic Medicine Institute Cleveland Clinic-Lerner Research Institute NE5-255 9620 Carnegie Ave Cleveland, OH 44106 work email: seyfim@ccf.orgmailto:seyfim@ccf.org mobile: (216)-633-9418 http://www.lerner.ccf.org/gmi/people/

On Jun 5, 2018, at 1:25 PM, Tim Sterne-Weiler notifications@github.com<mailto:notifications@github.com> wrote:

Closed #68https://github.com/timbitz/Whippet.jl/issues/68.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/timbitz/Whippet.jl/issues/68#event-1664286410, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AikLJdhzwHfDPgQvT6MfAIBhXuBG-41Bks5t5r72gaJpZM4UEsIu.

===================================

Please consider the environment before printing this e-mail

Cleveland Clinic is currently ranked as the No. 2 hospital in the country by U.S. News & World Report (2017-2018). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.