search

timbitz / Whippet.jl

Lightweight and Fast; RNA-seq quantification at the event-level
MIT License
103 stars 21 forks source link

Missing dependencies for Julia #138

Open astrophys opened 1 year ago

astrophys commented 1 year ago

Hello,

I installed julia for Red Hat 8.4 Linux. When I tried running Whippet, e.g.

whippet-index.jl --fasta /reference/homo_sapiens/GRCh38/ensembl/release-96/Sequence/WholeGenomeFasta/Homo_sapiens.GRCh38.dna.primary_assembly.fa --gtf /export/apps/opt/Whippet/1.6.1/anno/refseq_hg19.flat.gz

I got multiple errors complaining about missing Julia packages. In ~/.julia/packages I've installed 

/home/group/user/.julia/packages:
Automa         BioGenerics   BioSymbols  IntervalTrees  SIMD         TranscodingStreams
BioAlignments  BioSequences  FASTX       ScanByte       StringViews  Twiddle

Now when I run whippet-index.jl --fasta /reference/homo_sapiens/GRCh38/ensembl/release-96/Sequence/WholeGenomeFasta/Homo_sapiens.GRCh38.dna.primary_assembly.fa --gtf /export/apps/opt/Whippet/1.6.1/anno/refseq_hg19.flat.gz, I get 

$ whippet-index.jl --fasta /reference/homo_sapiens/GRCh38/ensembl/release-96/Sequence/WholeGenomeFasta/Homo_sapiens.GRCh38.dna.primary_assembly.fa --gtf /export/apps/opt/Whippet/1.6.1/anno/refseq_hg19.flat.gz
Whippet v1.6.1 loading...
  Activating project at `/export/apps/opt/Whippet/1.6.1`
ERROR: LoadError: ArgumentError: Package FASTX does not have StringViews in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have FASTX checked out for development and have
  added StringViews as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with FASTX
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1167 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include
   @ ./Base.jl:419 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1554
 [6] top-level scope
   @ stdin:1
in expression starting at /home/group/user/.julia/packages/FASTX/SLQYz/src/FASTX.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile FASTX [c2308a5c-f048-11e8-3e8a-31650f418d12] to /home/group/user/.julia/compiled/v1.8/FASTX/jl_VxqiDM.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1707
  [3] compilecache
    @ ./loading.jl:1651 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include
    @ ./Base.jl:419 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1554
 [11] top-level scope
    @ stdin:1
in expression starting at /export/apps/opt/Whippet/1.6.1/src/Whippet.jl:3
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Whippet [0af2a87d-76f3-44e3-af6e-8f91dcdf7847] to /home/group/user/.julia/compiled/v1.8/Whippet/jl_WCiFEP.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
in expression starting at /export/apps/opt/Whippet/1.6.1/bin/whippet-index.jl:14

I was under the impression that this would work 'out of the box', so I'm confused about where I'm going wrong.

Question :

  1. How do I install all the dependencies for Whippet?

Thanks!

timbitz commented 1 year ago

Hey @astrophys -- Sorry for the delay I just saw this. For reference, can I ask you what version of julia you are on? It looks like there is a new release 1.8.X since I last updated, and it looks like some packages changed dependencies. So it's just about updating the environment.

Not to be a stickler, but note that word in the readme should work 'out of the box'... wherein sometimes updates change dependencies of packages and newer versions of julia complain. Happy to update though!

astrophys commented 1 year ago

Fine point on "out of the box", I completely understand. I'm pretty naive with both this program and Julia, so I was thinking that it may be an operator issue on my end.

I'm using Julia 1.8.2.

Thank you for taking the time to respond and being willing to update.

timbitz commented 1 year ago

Hi @astrophys, Hmm-- I am able to build and pass tests successfully on the current master branch of Whippet with both julia 1.7.3 and julia 1.8.3, and I know that I was also successful previously with the current long-term release LTR 1.6.7.

Can you give me more information about what you are doing and try testing?

For testing, can you cd Whippet.jl and either try:

julia --project -e 'using Pkg; Pkg.instantiate(); Pkg.test()'
astrophys commented 1 year ago

Hello,

So I am not really sure 'what I'm trying to do.' I manage a compute resource and I'm installing this for one of our users. I ran the above command

julia --project -e 'using Pkg; Pkg.instantiate(); Pkg.test()'

Which seemed to complete successfully. I then reran :

whippet-index.jl --fasta /reference/homo_sapiens/GRCh38/ensembl/release-96/Sequence/WholeGenomeFasta/Homo_sapiens.GRCh38.dna.primary_assembly.fa --gtf /export/apps/opt/Whippet/1.6.1/anno/refseq_hg19.flat.gz

and encountered the error :

ERROR: LoadError: MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer
Stacktrace:
  [1] mapreduce_empty(#unused#::typeof(identity), op::Function, T::Type)
    @ Base ./reduce.jl:367
  [2] reduce_empty(op::Base.MappingRF{typeof(identity), typeof(min)}, #unused#::Type{UInt32})
    @ Base ./reduce.jl:356
  [3] reduce_empty_iter
    @ ./reduce.jl:379 [inlined]
  [4] mapreduce_empty_iter(f::Function, op::Function, itr::Vector{UInt32}, ItrEltype::Base.HasEltype)
    @ Base ./reduce.jl:375
  [5] _mapreduce
    @ ./reduce.jl:427 [inlined]
  [6] _mapreduce_dim
    @ ./reducedim.jl:365 [inlined]
  [7] #mapreduce#765
    @ ./reducedim.jl:357 [inlined]
  [8] mapreduce
    @ ./reducedim.jl:357 [inlined]
  [9] #_minimum#787
    @ ./reducedim.jl:999 [inlined]
 [10] _minimum
    @ ./reducedim.jl:999 [inlined]
 [11] #_minimum#786
    @ ./reducedim.jl:998 [inlined]
 [12] _minimum
    @ ./reducedim.jl:998 [inlined]
 [13] #minimum#784
    @ ./reducedim.jl:994 [inlined]
 [14] minimum
    @ ./reducedim.jl:994 [inlined]
 [15] (::Whippet.var"#private_add_transcript!#9"{Dict{String, Int64}, Dict{String, Float64}, Dict{String, Vector{RefTx}}, Dict{String, Tuple{Vararg{UInt32}}}, Dict{String, Tuple{Vararg{UInt32}}}, Dict{String, Tuple{Vararg{UInt32}}}, Dict{String, Tuple{Vararg{UInt32}}}, Dict{String, GeneInfo}})(curtran::String, curgene::String, curchrom::String, curstran::Char, trandon::Vector{UInt32}, tranacc::Vector{UInt32}, txlen::Int64)
    @ Whippet /gpfs0/export/apps/opt/Whippet/1.6.1/src/refset.jl:105
 [16] load_gtf(fh::BufferedStreams.BufferedInputStream{Libz.Source{:inflate, BufferedStreams.BufferedInputStream{IOStream}}}; txbool::Bool, suppress::Bool, usebam::Bool, bamreader::Nullable{XAM.BAM.Reader}, bamreads::Int64, bamoneknown::Bool)
    @ Whippet /gpfs0/export/apps/opt/Whippet/1.6.1/src/refset.jl:211
 [17] macro expansion
    @ /gpfs0/export/apps/opt/Whippet/1.6.1/src/timer.jl:5 [inlined]
 [18] main()
    @ Main /export/apps/opt/Whippet/1.6.1/bin/whippet-index.jl:91
 [19] top-level scope
    @ /gpfs0/export/apps/opt/Whippet/1.6.1/src/timer.jl:5
in expression starting at /export/apps/opt/Whippet/1.6.1/bin/whippet-index.jl:108

See attached output.

Thanks. whippet.txt

timbitz commented 1 year ago

Hey @astrophys. The unit tests cover almost all of whippet's functionality, so I would think your build is fine. What is probably not fine, is supplying the hg38 genome fasta file, with an hg19 gtf annotation file as it seems you are doing. My guess is those two don't match up and Whippet assumes that they should. Maybe try using the correct annotation for the genome build. A gencode hg38 gtf file should work with that genome build. To clean up the graph structure you should use the --suppress-low-tsl option in whippet-index.jl