search

meringlab / FlashWeave.jl

Inference of microbial interaction networks from large-scale heterogeneous abundance data
Other
74 stars 8 forks source link

KeyError: key FlashWeave [2be3f83a-7913-5748-9f20-7d448995b934] not found #27

Closed nick-youngblut closed 3 years ago

nick-youngblut commented 3 years ago

If I run the example code in a script on the HMP_SRA_gut_small.tsv dataset (with a couple of random test metadata columns: HMP_SRA_gut_small_meta.tsv), I get the following error when running learn_network() with >1 proc set via addprocs():

### Loading data ###

Inferring network with FlashWeave - sensitive (conditional)

    Run information:
    sensitive - true
    heterogeneous - false
    max_k - 3
    alpha - 0.01
    sparse - false
    workers - 4
    OTUs - 50
    MVs - 2

### Normalizing ###

Removing variables with 0 variance (or equivalently 1 level) and samples with 0 reads
Discarded 5 samples and 0 variables.

Normalization

### Learning interactions ###

Setting 'time_limit' to 6.0 s.
Automatically setting 'n_obs_min' to 20 for enhanced reliability.
Computing univariate associations..
ERROR: LoadError: On worker 2:
KeyError: key FlashWeave [2be3f83a-7913-5748-9f20-7d448995b934] not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:482 [inlined]
  [2] root_module
    @ ./loading.jl:957 [inlined]
  [3] deserialize_module
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:962
  [4] handle_deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:864
  [5] deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
  [6] deserialize_datatype
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1287
  [7] handle_deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:835
  [8] deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
  [9] handle_deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:842
 [10] deserialize
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782 [inlined]
 [11] deserialize_msg
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/messages.jl:87
 [12] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [13] invokelatest
    @ ./essentials.jl:706 [inlined]
 [14] message_handler_loop
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:169
 [15] process_tcp_streams
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:126
 [16] #99
    @ ./task.jl:411
Stacktrace:
  [1] #remotecall_fetch#143
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:394 [inlined]
  [2] remotecall_fetch(::Function, ::Distributed.Worker)
    @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:386
  [3] remotecall_fetch(::Function, ::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
  [4] remotecall_fetch
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421 [inlined]
  [5] workers_all_local()
    @ FlashWeave ~/.julia/packages/FlashWeave/464SQ/src/misc.jl:90
  [6] prepare_univar_results(data::Matrix{Float32}, test_name::String, alpha::Float64, hps::Int64, n_obs_min::Int64, FDR::Bool, levels::Vector{Int32}, parallel::String, cor_mat::Matrix{Float32}, correct_reliable_only::Bool, verbose::Bool, tmp_folder::String)
    @ FlashWeave ~/.julia/packages/FlashWeave/464SQ/src/learning.jl:86
  [7] LGL(data::Matrix{Float32}; test_name::String, max_k::Int64, alpha::Float64, hps::Int64, n_obs_min::Int64, max_tests::Int64, convergence_threshold::Float64, FDR::Bool, parallel::String, fast_elim::Bool, no_red_tests::Bool, weight_type::String, edge_rule::String, nonsparse_cond::Bool, verbose::Bool, update_interval::Float64, edge_merge_fun::typeof(FlashWeave.maxweight), tmp_folder::String, debug::Int64, time_limit::Float64, header::Vector{String}, meta_variable_mask::Nothing, dense_cor::Bool, recursive_pcor::Bool, cache_pcor::Bool, correct_reliable_only::Bool, feed_forward::Bool, track_rejections::Bool, all_univar_nbrs::Nothing, kill_remote_workers::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FlashWeave ~/.julia/packages/FlashWeave/464SQ/src/learning.jl:229
  [8] macro expansion
    @ ./timing.jl:368 [inlined]
  [9] learn_network(data::Matrix{Any}; sensitive::Bool, heterogeneous::Bool, max_k::Int64, alpha::Float64, conv::Float64, header::Vector{String}, meta_mask::BitVector, feed_forward::Bool, fast_elim::Bool, normalize::Bool, track_rejections::Bool, verbose::Bool, transposed::Bool, prec::Int64, make_sparse::Bool, make_onehot::Bool, max_tests::Int64, hps::Int64, FDR::Bool, n_obs_min::Int64, cache_pcor::Bool, time_limit::Float64, update_interval::Float64, parallel_mode::String)
    @ FlashWeave ~/.julia/packages/FlashWeave/464SQ/src/learning.jl:447
 [10] learn_network(data_path::String, meta_data_path::String; otu_data_key::String, otu_header_key::String, meta_data_key::String, meta_header_key::String, verbose::Bool, transposed::Bool, kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol, Symbol}, NamedTuple{(:sensitive, :heterogeneous), Tuple{Bool, Bool}}})
    @ FlashWeave ~/.julia/packages/FlashWeave/464SQ/src/learning.jl:316
 [11] top-level scope
    @ /ebio/abt3_projects/software/dev/FlashWeave.jl/sandbox.jl:48
in expression starting at /ebio/abt3_projects/software/dev/FlashWeave.jl/sandbox.jl:48

This appears to occur if I call addprocs() after using FlashWeave

jtackm commented 3 years ago

Yes, this error comes from the Distributed package and is "by design". If you addprocs with that package, the added workers don't automatically inherit the parent environment (including all packages), but have to explicitly be asked to load packages themselves. This is mildly annoying and will hopefully be changed at some point.