search

kharchenkolab / Baysor

Bayesian Segmentation of Spatial Transcriptomics Data
https://kharchenkolab.github.io/Baysor/
MIT License
155 stars 31 forks source link

Precompilation warnings during Baysor run #92

Closed hchen725 closed 2 months ago

hchen725 commented 1 year ago

Hi! Thanks for putting together an awesome package and algorithm. I'm not super familiar with Julia so my apologies if this is something super trivial.

I've been running Baysor on multiple regions and I sometimes get a warning error regarding precompiled cache files that look something like this:

[21:43:22] Warning: The call to compilecache failed to create a usable precompiled cache file for Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0]
└ Base loading.jl:1818
[21:44:20] Warning: The call to compilecache failed to create a usable precompiled cache file for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
└ Base loading.jl:1818
[21:45:07] Warning: The call to compilecache failed to create a usable precompiled cache file for NearestNeighbors [b8a86587-4115-5ab1-83bc-aa920d37bbce]
└ Base loading.jl:1818

It's been super variable across Baysor runs and for a different Baysor run, I might get a precompilation cache warning for CSV instead, might error out...or might run perfectly.

When it completely errors out, using the Pkg.recompile() function and that seems to be ok, but was wondering if there's other ways to patch this error. I'd like to incorporate it into a larger workflow so would love to understand the causes of these various errors/warnings. Thanks again!

hchen725 commented 1 year ago

An example of when warnings arise but the run successfully completes:

┌ Warning: Module Parsers with build ID fafbfcfd-bfd8-68b1-0007-33d20f1006d3 is missing from the cache.
│ This may mean Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module DataAPI with build ID fafbfcfd-5997-3de8-001a-8578660a0cc3 is missing from the cache.
│ This may mean DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module PooledArrays with build ID fafbfcfd-d0f1-6a9d-001a-8579e4410e98 is missing from the cache.
│ This may mean PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Tables with build ID fafbfcfd-3dd2-cf15-001a-8d53c112f616 is missing from the cache.
│ This may mean Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module DataAPI with build ID fafbfcfd-5997-3de8-001a-8578660a0cc3 is missing from the cache.
│ This may mean DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Parsers with build ID fafbfcfd-bfd8-68b1-0007-33d20f1006d3 is missing from the cache.
│ This may mean Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module StatsBase with build ID ffffffff-ffff-ffff-0007-33dddbed6324 is missing from the cache.
│ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module SpecialFunctions with build ID ffffffff-ffff-ffff-0007-33e54abda50c is missing from the cache.
│ This may mean SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module SpecialFunctions with build ID ffffffff-ffff-ffff-0007-33e54abda50c is missing from the cache.
│ This may mean SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module DataStructures with build ID fafbfcfd-f5c3-2260-0007-33d9f9097a77 is missing from the cache.
│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
WARNING: method definition for InitialParams at /wynton/home/rotation/hchen95/.conda/envs/merfish-snakemake/share/julia/packages/Baysor/rFTAp/src/processing/models/InitialParams.jl:6 declares type variable N but does not use it.
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module JLLWrappers with build ID ffffffff-ffff-ffff-0007-33dffa5f805d is missing from the cache.
│ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Distributions with build ID ffffffff-ffff-ffff-0007-33e428e0ed75 is missing from the cache.
│ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Preferences with build ID fafbfcfd-2a71-28a4-001a-8da1b7073637 is missing from the cache.
│ This may mean Preferences [21216c6a-2e73-6563-6e65-726566657250] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module ForwardDiff with build ID ffffffff-ffff-ffff-0007-33ef076ab4b9 is missing from the cache.
│ This may mean ForwardDiff [f6369f11-7733-5829-9624-2563aa707210] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module DataStructures with build ID fafbfcfd-f5c3-2260-0007-33d9f9097a77 is missing from the cache.
│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Compat with build ID fafbfcfd-fc1c-d936-001a-857d57499ce8 is missing from the cache.
│ This may mean Compat [34da2185-b29b-5c13-b0c7-acf172513d20] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module StatsBase with build ID ffffffff-ffff-ffff-0007-33dddbed6324 is missing from the cache.
│ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module SpecialFunctions with build ID ffffffff-ffff-ffff-0007-33e54abda50c is missing from the cache.
│ This may mean SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module StatsFuns with build ID ffffffff-ffff-ffff-0007-33e4d9070cef is missing from the cache.
│ This may mean StatsFuns [4c63d2b9-4356-54db-8cca-17b64c39e42c] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Distributions with build ID ffffffff-ffff-ffff-0007-33e428e0ed75 is missing from the cache.
│ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793

An example when warnings arise causing an execution error and the run stops:

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0]
│   exception = Required dependency Preferences [21216c6a-2e73-6563-6e65-726566657250] failed to load from a cache file.
└ @ Base loading.jl:1818
┌ Warning: Module Parsers with build ID ffffffff-ffff-ffff-001a-8d205a86d8c5 is missing from the cache.
│ This may mean Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Parsers with build ID ffffffff-ffff-ffff-001a-8d205a86d8c5 is missing from the cache.
│ This may mean Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Compat with build ID fafbfcfd-7a8a-a542-001a-8dbba40f52d6 is missing from the cache.
│ This may mean Compat [34da2185-b29b-5c13-b0c7-acf172513d20] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Compat with build ID fafbfcfd-7a8a-a542-001a-8dbba40f52d6 is missing from the cache.
│ This may mean Compat [34da2185-b29b-5c13-b0c7-acf172513d20] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Compat with build ID fafbfcfd-7a8a-a542-001a-8dbba40f52d6 is missing from the cache.
│ This may mean Compat [34da2185-b29b-5c13-b0c7-acf172513d20] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module DataStructures with build ID ffffffff-ffff-ffff-001a-8d26e49089eb is missing from the cache.
│ This may mean DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module StatsAPI with build ID fafbfcfd-9c9f-d7ce-001a-8dcc77909ae4 is missing from the cache.
│ This may mean StatsAPI [82ae8749-77ed-4fe6-ae5f-f523153014b0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Compat with build ID fafbfcfd-7a8a-a542-001a-8dbba40f52d6 is missing from the cache.
│ This may mean Compat [34da2185-b29b-5c13-b0c7-acf172513d20] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Parsers with build ID ffffffff-ffff-ffff-001a-8d205a86d8c5 is missing from the cache.
│ This may mean Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module StatsBase with build ID ffffffff-ffff-ffff-001a-8d337761af77 is missing from the cache.
│ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module Reexport with build ID fafbfcfd-637c-9fdd-001a-8dbfa2b50df7 is missing from the cache.
│ This may mean Reexport [189a3867-3050-52da-a836-e630ba90ab69] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
┌ Warning: Module IrrationalConstants with build ID fafbfcfd-7a0f-b8af-001a-8dcb9c55c0b6 is missing from the cache.
│ This may mean IrrationalConstants [92d709cd-6900-40b7-9082-c6be49f344b6] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1793
ERROR: LoadError: Unable to find compatible target in system image.
Stacktrace:
  [1] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1052
  [2] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [3] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [4] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
  [5] macro expansion
    @ ./loading.jl:1648 [inlined]
  [6] macro expansion
    @ ./lock.jl:267 [inlined]
  [7] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
  [8] include
    @ ./Base.jl:457 [inlined]
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1840
 [10] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [11] macro expansion
    @ ./loading.jl:1648 [inlined]
 [12] macro expansion
    @ ./lock.jl:267 [inlined]
 [13] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
 [14] include
    @ ./Base.jl:457 [inlined]
 [15] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1840
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [17] macro expansion
    @ ./loading.jl:1648 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
 [20] include
    @ ./Base.jl:457 [inlined]
 [21] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1840
 [22] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [23] macro expansion
    @ ./loading.jl:1648 [inlined]
 [24] macro expansion
    @ ./lock.jl:267 [inlined]
 [25] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
 [26] include(mod::Module, _path::String)
    @ Base ./Base.jl:457
 [27] include(x::String)
    @ Baysor ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/Baysor.jl:1
 [28] top-level scope
    @ none:1
 [29] eval
    @ ./boot.jl:370 [inlined]
 [30] (::Baysor.LazySubmodules.var"#1#2"{Baysor.LazySubmodules.LazySubmodule})()
    @ Baysor.LazySubmodules ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/LazySubmodules.jl:33
 [31] lock(f::Baysor.LazySubmodules.var"#1#2"{Baysor.LazySubmodules.LazySubmodule}, l::ReentrantLock)
    @ Base ./lock.jl:229
 [32] load(m::Baysor.LazySubmodules.LazySubmodule)
    @ Baysor.LazySubmodules ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/LazySubmodules.jl:31
 [33] getproperty(m::Baysor.LazySubmodules.LazySubmodule, s::Symbol)
    @ Baysor.LazySubmodules ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/LazySubmodules.jl:62
 [34] load_prior_segmentation!(df_spatial::DataFrames.DataFrame, prior_segmentation::String, opts::Baysor.Utils.SegmentationOptions; min_molecules_per_cell::Int64, min_molecules_per_segment::Int64, plot::Bool)
    @ Baysor.CommandLine ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/cli/main.jl:197
 [35] run(coordinates::String, prior_segmentation::String; config::Baysor.Utils.RunOptions, x_column::String, y_column::String, z_column::String, gene_column::String, min_molecules_per_cell::Int64, scale::Float64, scale_std::String, n_clusters::Int64, prior_segmentation_confidence::Float64, output::String, plot::Bool, save_polygons::String, no_ncv_estimation::Bool, count_matrix_format::String)
    @ Baysor.CommandLine ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/cli/main.jl:108
 [36] command_main(ARGS::Vector{String})
    @ Baysor.CommandLine ~/.julia/packages/Comonicon/HDhA6/src/codegen/julia.jl:343
 [37] command_main()
    @ Baysor.CommandLine ~/.julia/packages/Comonicon/HDhA6/src/codegen/julia.jl:90
 [38] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [39] invokelatest
    @ ./essentials.jl:813 [inlined]
 [40] #6#7
    @ ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/LazySubmodules.jl:70 [inlined]
 [41] (::Baysor.LazySubmodules.var"#6#9"{Baysor.LazySubmodules.var"#6#7#10"{typeof(Baysor.CommandLine.command_main)}})()
    @ Baysor.LazySubmodules ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/LazySubmodules.jl:70
 [42] command_main(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Baysor ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/Baysor.jl:41
 [43] command_main()
    @ Baysor ~/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/Baysor.jl:41
 [44] top-level scope
    @ ~/projects/merfish-crc/src/baysor:15
in expression starting at /User/hchen/.julia/packages/SpecialFunctions/sKqs4/src/SpecialFunctions.jl:1
in expression starting at /User/hchen/.julia/packages/StatsFuns/atA5S/src/StatsFuns.jl:1
in expression starting at /User/hchen/.julia/packages/Distributions/GrN7f/src/Distributions.jl:1
in expression starting at /User/hchen/.conda/envs/snakemake-env/share/julia/packages/Baysor/rFTAp/src/processing/Processing.jl:1
in expression starting at /User/hchen/projects/project-name/src/baysor:15
VPetukhov commented 2 months ago

That's a julia thing. I think they improved it in the newer releases.