search

JuliaRobotics / RoME.jl

Robot Motion Estimate: Tools, Variables, and Factors for SLAM in robotics; also see Caesar.jl.
MIT License
64 stars 13 forks source link

Error running example using multiple processes #407

Closed lemauee closed 3 years ago

lemauee commented 3 years ago

Hi, I just wanted to run the example from #396 using multiple processes and got the following error:

ERROR: LoadError: TaskFailedException:
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#432#435"{MetaBayesTree,Bool,Bool,Base.TTY,Int64,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64})() at ./threadingconstructs.jl:169
Stacktrace:
 [1] [ Info: monitorCSMs: all tasks done
sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:371
 [5] top-level scope at /home/leopold/Nextcloud/Studium/Masterarbeit/svn/julia/mmiSAM/examples/speed/speed.jl:118
 [6] top-level scope at timing.jl:233
 [7] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088
 [8] include_string(::Module, ::String, ::String) at ./loading.jl:1096
 [9] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./essentials.jl:710
 [10] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:709
 [11] inlineeval(::Module, ::String, ::Int64, ::Int64, ::String; softscope::Bool) at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:185
 [12] (::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams})() at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:144
 [13] withpath(::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams}, ::String) at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:124
 [14] (::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:142
 [15] hideprompt(::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams}) at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/repl.jl:36
 [16] (::VSCodeServer.var"#59#63"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:110
 [17] with_logstate(::Function, ::Any) at ./logging.jl:408
 [18] with_logger at ./logging.jl:514 [inlined]
 [19] (::VSCodeServer.var"#58#62"{VSCodeServer.ReplRunCodeRequestParams})() at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:109
 [20] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [21] invokelatest(::Any) at ./essentials.jl:709
 [22] macro expansion at /home/leopold/.vscode-oss/extensions/julialang.language-julia-1.0.10/scripts/packages/VSCodeServer/src/eval.jl:27 [inlined]
 [23] (::VSCodeServer.var"#56#57")() at ./task.jl:356
in expression starting at /home/leopold/Nextcloud/Studium/Masterarbeit/svn/julia/mmiSAM/examples/speed/speed.jl:

The only thing I did in the REPL before executing the Code from #396 was

using Distributed
addprocs(3)

on single process it runs fine

Best, Leo

dehann commented 3 years ago

Hi Leo,

Oops that looks unfamiliar, assuming you doing something similar to this -- I found the following sequence is the most reliable (without using Revise.jl):

using Distributed
using RoME
addprocs(3)
using RoME
@everywhere RoME 

# some local precompiling to speed up later usage
warmUpSolverJIT()

Would you mind quickly trying this sequence on that example please and let us know? If that does not do it and something new is happening, the fastest place to debug looks like area of the code called CSM (CliqStateMachine).

Best, Dehann

lemauee commented 3 years ago

Hi Dehann,

First off I found a typo, I assume you meant @everywhere using RoME and not @everywhere RoME :)

Now to the long story for the error, I'll post a tl;dr version below:

VSCode uses Revise.jl by default (which Juno did not do by default), when running @everywhere using RoME it already has compilation errors I did not put much interest in until now:

julia> @everywhere using RoME
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 3:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 4:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 2:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME

The warmUpSolverJIT() then fails with a similar error as my example

[ Info: do init of x6
[ Info: try doautoinit! of l1
[ Info: init with useinitfct [:x0l1f1, :x6l1f1]
[ Info: do init of l1
[ Info: [2x0p,d2,N100],
[ Info: Ensure variables are all initialized (graphinit)
[ Info: Solving over the Bayes (Junction) tree.
Building Bayes net...
Find potential functions for each clique
[ Info: Bayes Tree Complete
[ Info: Do tree based init-inference on tree
[ Info: Solve is drawing the Bayes tree
Solve Progress: approx max 144, at iter 9        Time: 0:00:05
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:

RemoteException(3, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))RemoteException(4, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:
Stacktrace:error
( [1] error
( [1] error::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] 
approxCliqMarginalUp!RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))::(
Stacktrace:RemoteException):: at CliqStateMachineContainer./error.jl:42{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, 
 [1] error
( [2] approxCliqMarginalUp!::(Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::RemoteException):: at CliqStateMachineContainer./error.jl:42{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, N::Int64N, ::Int64
,  [2] approxCliqMarginalUp!::(Array{LikelihoodMessage,1}; dbg::Booldbg, ::Bool::, CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, N::Int64multiproc, ::Boolmultiproc, ::Bool::, Array{LikelihoodMessage,1}; dbg::Boollogger, ::Base.CoreLogginglogger.::SimpleLoggerBase.CoreLogging, N.::SimpleLoggerInt64, multiproc, ::Booliters, ::Int64iters, ::Int64dbg, ::Boollogger, ::Base.CoreLoggingdrawpdf.::SimpleLoggerBool, drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379multiproc):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379iters, ::Int64
,  [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!logger(::Base.CoreLoggingdrawpdf.::SimpleLoggerBool, ::)CliqStateMachineContainer at {/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})iters at ::/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30Int64
,  [4] solveUp_StateMachine
( [3] __doCliqUpSolveInitialized!
( [4] solveUp_StateMachinedrawpdf(::Bool::)CliqStateMachineContainer at {/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [3] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333__doCliqUpSolveInitialized!
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [4] solveUp_StateMachine::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [4] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333solveUp_StateMachine
( [5] (::StateMachine{CliqStateMachineContainer})::(Nothing; ::Nothing; ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, pollinterval::Float64pollinterval, ::Float64
,  [5] (::StateMachine{CliqStateMachineContainer})::(Nothing; breakafter::Functionbreakafter, ::Function::, CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, pollinterval::Float64verbose, ::Boolverbose, ::Bool::, Nothing; breakafter::Functionverbosefid, ::Baseverbosefid.::TTYBase, pollinterval.::TTYFloat64, verbose, ::BoolverboseXtra, ::IncrementalInferenceverboseXtra.::CliqStatusIncrementalInference, breakafter.::CliqStatusFunction, verbosefid, ::Baseiterlimit.::TTYInt64, iterlimit, ::Int64verbose, ::BoolverboseXtra, ::IncrementalInferenceinjectDelayBefore.::CliqStatusNothing, injectDelayBefore, ::Nothingverbosefid, ::Baseiterlimit.::TTYInt64, recordhistory, ::Boolrecordhistory, ::BoolverboseXtra, ::IncrementalInferenceinjectDelayBefore.::CliqStatusNothing, housekeeping_cb, ::IncrementalInferencehousekeeping_cb.::IncrementalInferenceiterlimit.::Int64recordhistory, ::Boolvar, "#377#379"{varIncrementalInference"#377#379".{TreeCliqueinjectDelayBeforeIncrementalInference}::.)NothingTreeClique at housekeeping_cb, }/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82::)IncrementalInference at recordhistory./home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82::Bool
,  [6] initStartCliqStateMachine!var("#377#379"{
IncrementalInference [6] .initStartCliqStateMachine!TreeCliquehousekeeping_cb(}::)IncrementalInference at ::./home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82LightDFG{SolverParams,DFGVariable,DFGFactor}, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, var"#377#379"{
IncrementalInference [6] .initStartCliqStateMachine!TreeClique::(}MetaBayesTree),  at ::/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::IncrementalInference.TreeClique, 
 [6] initStartCliqStateMachine!::(IncrementalInference.TreeClique, ::MetaBayesTree, ::Nothing; ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Nothing; ::IncrementalInference.TreeClique, oldcliqdata::BayesTreeNodeData::, MetaBayesTree, oldcliqdata::BayesTreeNodeData::, Nothing; verbose::Bool::, IncrementalInference.TreeClique, verbose::Boololdcliqdata, ::BayesTreeNodeDataverbosefid, ::Base::.NothingTTY; , verbosefid::Baseverbose.::TTYBool, oldcliqdata, ::BayesTreeNodeDatadrawtree, ::Booldrawtree, ::Boolverbosefid, ::Baseverbose.::TTYBool, show, ::Boolshow, ::Booldrawtree, ::Boolverbosefid, ::Baseincremental.::TTYBool, incremental, ::Boolshow, ::Booldrawtree, ::Boollimititers, ::Int64limititers, ::Int64incremental, ::Boolshow, ::Boolupsolve, ::Boolupsolve, ::Boollimititers, ::Int64incremental, ::Booldownsolve, ::Booldownsolve, ::Boolupsolve, ::Boollimititers, ::Int64recordhistory, ::Boolrecordhistory, ::Booldownsolve, ::Boolupsolve, ::Booldelay, ::Booldelay, ::Boolrecordhistory, ::Booldownsolve, ::Boollogger, ::Base.CoreLogginglogger.::SimpleLoggerBase.CoreLogging, delay.::SimpleLoggerBool, recordhistory, ::Boolsolve_progressbar, ::ProgressMetersolve_progressbar.::ProgressUnknownProgressMeter, logger.::ProgressUnknownBase.CoreLogging, delay.::SimpleLoggerBool, algorithm, ::Symbolalgorithm):: at Symbol/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63logger.::ProgressUnknownBase.CoreLogging, 
. [7] SimpleLoggertryCliqStateMachineSolve!, 
( [7] tryCliqStateMachineSolve!algorithm(::Symbolsolve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63::.LightDFGProgressUnknown{, SolverParams,DFGVariable,DFGFactor}, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!algorithm(::Symbol::)MetaBayesTree at , /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Int64, 
 [7] tryCliqStateMachineSolve!::(Int64, ::MetaBayesTree, ::Nothing; ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Nothing; ::Int64, oldtree::MetaBayesTree::, MetaBayesTree, oldtree::MetaBayesTree::, Nothing; verbose::Bool::, Int64, verbose::Boololdtree, ::MetaBayesTreeverbosefid, ::Base::.NothingTTY; , verbosefid::Baseverbose.::TTYBool, oldtree, ::MetaBayesTreedrawtree, ::Booldrawtree, ::Boolverbosefid, ::Baseverbose.::TTYBool, limititers, ::Int64limititers, ::Int64drawtree, ::Boolverbosefid, ::Basedownsolve.::TTYBool, downsolve, ::Boollimititers, ::Int64drawtree, ::Boolincremental, ::Boolincremental, ::Booldownsolve, ::Boollimititers, ::Int64delaycliqs, ::Arraydelaycliqs{::SymbolArray,incremental{1::Symbol}Bool,, downsolve, 1::}Bool, recordcliqs, ::Arraydelaycliqs{::SymbolArray,recordcliqs{1::Symbol}Array,, incremental{1::Symbol}Bool,, solve_progressbar, 1::}ProgressMeter, recordcliqs.::ProgressUnknownArray, delaycliqs{::SymbolArray,solve_progressbar{1::Symbol}ProgressMeter,, algorithm.1::ProgressUnknown}Symbol, , solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110algorithm.::ProgressUnknownSymbol, recordcliqs):: at Array/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
{ [8] Symbol(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64}),algorithm(1::)}Symbol at , 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
solve_progressbar(::)ProgressMeter at 
../task.jl:356 [8] ProgressUnknown┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64}), ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(algorithm):: at Symbol./task.jl:356)┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
 at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Error: Task 3 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
[ Info: All cliques should have exited
┌ Error: Task 4 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
Solve Progress: approx max 144, at iter 21       Time: 0:00:08┌ Error: CSM clique 1: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
Solve Progress: approx max 144, at iter 22       Time: 0:00:09[ Info: All cliques should have exited

When turning off Revise.jl loading, I get the same compilation errors as above, and the same errors appear when running warmUpSolverJIT()

julia> using Distributed
julia> using RoME
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
[ Info: Including InteractiveUtils related functions in IncrementalInference.

julia> addprocs(3)
3-element Array{Int64,1}:
 2
 3
 4

julia> using RoME

julia> @everywhere using RoME
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 3:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 4:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 2:    │   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
│   exception = Required dependency Optim [429524aa-4258-5aef-a3af-852621145aeb] failed to load from a cache file.
└ @ Base loading.jl:1042
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME

julia> warmUpSolverJIT()
[ Info: try doautoinit! of x0
STEP 1: Driving around a bit
[ Info: try doautoinit! of x0
[ Info: init with useinitfct [:x0f1]
[ Info: do init of x0
┌ Warning: updateVariableSolverData with solveKey parameter might change in the future, see DFG #565. Future warnings are suppressed
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/services/DFGVariable.jl:526
[ Info: try doautoinit! of x1
[ Info: try doautoinit! of x1
[ Info: init with useinitfct [:x0x1f1]
[ Info: do init of x1
[ Info: try doautoinit! of x2
[ Info: try doautoinit! of x2
[ Info: init with useinitfct [:x1x2f1]
[ Info: do init of x2
[ Info: try doautoinit! of x3
[ Info: try doautoinit! of x3
[ Info: init with useinitfct [:x2x3f1]
[ Info: do init of x3
[ Info: try doautoinit! of x4
[ Info: try doautoinit! of x4
[ Info: init with useinitfct [:x3x4f1]
[ Info: do init of x4
[ Info: try doautoinit! of x5
[ Info: try doautoinit! of x5
[ Info: init with useinitfct [:x4x5f1]
[ Info: do init of x5
[ Info: try doautoinit! of x6
[ Info: try doautoinit! of l1
[ Info: try doautoinit! of x6
[ Info: init with useinitfct [:x5x6f1]
[ Info: do init of x6
[ Info: try doautoinit! of l1
[ Info: init with useinitfct [:x0l1f1, :x6l1f1]
[ Info: do init of l1
[ Info: [2x0p,d2,N100],
[ Info: Ensure variables are all initialized (graphinit)
[ Info: Solving over the Bayes (Junction) tree.
Building Bayes net...
Find potential functions for each clique
[ Info: Bayes Tree Complete
[ Info: Do tree based init-inference on tree
[ Info: Solve is drawing the Bayes tree
Solve Progress: approx max 144, at iter 9        Time: 0:00:05
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:

RemoteException(3, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))RemoteException(4, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:
Stacktrace:error

(RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:error
( [1] error
( [1] error::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; N::Int64N, ::Int64N, ::Int64N, ::Int64dbg, ::Booldbg, ::Booldbg, ::Booldbg, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boollogger, ::Base.CoreLogginglogger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, iters.::SimpleLoggerInt64, iters, ::Int64iters, ::Int64iters, ::Int64drawpdf, ::Booldrawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
) [3]  at __doCliqUpSolveInitialized!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [4] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [5] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333(::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; ::Nothing; ::Nothing; ::Nothing; pollinterval::Float64pollinterval, ::Float64pollinterval, ::Float64pollinterval, ::Float64breakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionverbose, ::Boolverbose, ::Boolverbose, ::Boolverbose, ::Boolverbosefid, ::Baseverbosefid.::TTYBase, verbosefid.::TTYBase, verbosefid.::TTYBase, verboseXtra.::TTYIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, iterlimit.::CliqStatusInt64, iterlimit, ::Int64iterlimit, ::Int64iterlimit, ::Int64injectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::Nothingrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolhousekeeping_cb, ::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencevar."#377#379"{varIncrementalInference"#377#379".{TreeCliquevarIncrementalInference}"#377#379".){TreeClique at varIncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82"#377#379".){TreeClique at 
IncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] .)initStartCliqStateMachine!TreeClique at 
(}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] )initStartCliqStateMachine! at ::(/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::Nothing; ::IncrementalInference.TreeClique, ::Nothing; ::IncrementalInference.TreeClique, oldcliqdata::BayesTreeNodeData::, Nothing; oldcliqdata::BayesTreeNodeData::, Nothing; verbose::Boololdcliqdata, ::BayesTreeNodeDataverbose, ::Boololdcliqdata, ::BayesTreeNodeDataverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baseshow.::TTYBool, drawtree, ::Boolshow, ::Booldrawtree, ::Boolincremental, ::Boolshow, ::Boolincremental, ::Boolshow, ::Boollimititers, ::Int64incremental, ::Boollimititers, ::Int64incremental, ::Boolupsolve, ::Boollimititers, ::Int64upsolve, ::Boollimititers, ::Int64downsolve, ::Boolupsolve, ::Booldownsolve, ::Boolupsolve, ::Boolrecordhistory, ::Booldownsolve, ::Boolrecordhistory, ::Booldownsolve, ::Booldelay, ::Boolrecordhistory, ::Booldelay, ::Boolrecordhistory, ::Boollogger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, logger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, solve_progressbar, ::ProgressMeterlogger.::ProgressUnknownBase.CoreLogging, solve_progressbar.::SimpleLoggerProgressMeter, logger.::ProgressUnknownBase.CoreLogging, algorithm.::SimpleLoggerSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
. [7] ProgressUnknowntryCliqStateMachineSolve!, algorithm(::Symbol
) [7]  at tryCliqStateMachineSolve!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm(::Symbol::)LightDFG at {/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Int64, ::MetaBayesTree, ::Int64, ::MetaBayesTree, ::Nothing; ::Int64, ::Nothing; ::Int64, oldtree::MetaBayesTree::, Nothing; oldtree::MetaBayesTree::, Nothing; verbose::Boololdtree, ::MetaBayesTreeverbose, ::Boololdtree, ::MetaBayesTreeverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baselimititers.::TTYInt64, drawtree, ::Boollimititers, ::Int64drawtree, ::Booldownsolve, ::Boollimititers, ::Int64downsolve, ::Boollimititers, ::Int64incremental, ::Booldownsolve, ::Boolincremental, ::Booldownsolve, ::Booldelaycliqs, ::Arrayincremental{::SymbolBool,delaycliqs, 1::}Array, incremental{::SymbolBool,delaycliqs, 1::}Array, recordcliqs{::SymbolArray,delaycliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, solve_progressbar{1::Symbol}ProgressMeter,, recordcliqs.1::ProgressUnknown}Array, , solve_progressbar{::SymbolProgressMeter,algorithm.1::ProgressUnknown}Symbol, , solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
. [8] ProgressUnknown(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64}), algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(
) [8]  at (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})./task.jl:356(┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
)┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
 at ./task.jl:356┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Error: Task 3 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
[ Info: All cliques should have exited
┌ Error: Task 4 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
Solve Progress: approx max 144, at iter 21       Time: 0:00:10┌ Error: CSM clique 1: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
Solve Progress: approx max 144, at iter 22       Time: 0:00:10[ Info: All cliques should have exited
┌ Error: Task 5 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
┌ Error: CSM clique 2: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
[ Info: All cliques should have exited
┌ Error: Task 6 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
ERROR: TaskFailedException:
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333[ Info: All cliques should have exited
 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:371
 [5] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:280 (repeats 2 times)
 [6] warmUpSolverJIT(; fg::LightDFG{SolverParams,DFGVariable,DFGFactor}, drawtree::Bool) at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:231
 [7] warmUpSolverJIT() at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:226
 [8] top-level scope at REPL[6]:1

julia> [ Info: monitorCSMs: all tasks done

Even updating from julia 1.5.2 to 1.5.3 did not make anything work out better:

[ Info: Precompiling VSCodeServer [9f5989ce-84fe-42d4-91ec-6a7a8d53ed0f]
julia> using Distributed

julia> using RoME
[ Info: Precompiling RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
[ Info: Including InteractiveUtils related functions in IncrementalInference.

julia> addprocs(3)
3-element Array{Int64,1}:
 2
 3
 4

julia> using RoME

julia> @everywhere using RoME
      From worker 3:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    ┌ Warning: Module DataAPI with build ID 3314943043456 is missing from the cache.
      From worker 4:    │ This may mean DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 3:    ┌ Warning: Module DataAPI with build ID 3314937261619 is missing from the cache.
      From worker 3:    │ This may mean DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a] does not support precompilation but is imported by a module that does.
      From worker 3:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 4:    │   exception = Required dependency SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 2:    │   exception = Invalid input in module list: expected Profile.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
      From worker 3:    │   exception = Invalid input in module list: expected Profile.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Distributions [31c24e10-a181-5473-b8eb-7969acd0382f]
      From worker 2:    │   exception = Required dependency QuadGK [1fd47b50-473d-5c70-9696-f719f8f3bcdc] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Distributions [31c24e10-a181-5473-b8eb-7969acd0382f]
      From worker 4:    │   exception = Invalid input in module list: expected QuadGK.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Distributions [31c24e10-a181-5473-b8eb-7969acd0382f]
      From worker 3:    │   exception = Required dependency QuadGK [1fd47b50-473d-5c70-9696-f719f8f3bcdc] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: Module Distributions with build ID 3326548195295 is missing from the cache.
      From worker 2:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: Module Distributions with build ID 3326595766356 is missing from the cache.
      From worker 4:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 3:    ┌ Warning: Module Distributions with build ID 3326647048939 is missing from the cache.
      From worker 3:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 3:    └ @ Base loading.jl:1017
      From worker 2:    ┌ Warning: Module KernelDensityEstimate with build ID 3331963940288 is missing from the cache.
      From worker 2:    │ This may mean KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: Module KernelDensityEstimate with build ID 3332029497397 is missing from the cache.
      From worker 4:    │ This may mean KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 3:    ┌ Warning: Module KernelDensityEstimate with build ID 3332039469810 is missing from the cache.
      From worker 3:    │ This may mean KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] does not support precompilation but is imported by a module that does.
      From worker 3:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for NLsolve [2774e3e8-f4cf-5e23-947b-6d7e65073b56]
      From worker 4:    │   exception = Required dependency Distances [b4f34e82-e78d-54a5-968a-f98e89d6e8f7] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: Module StatsBase with build ID 3326820657739 is missing from the cache.
      From worker 2:    │ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: Module StatsBase with build ID 3326857529968 is missing from the cache.
      From worker 4:    │ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Optim [429524aa-4258-5aef-a3af-852621145aeb]
      From worker 3:    │   exception = Required dependency DiffEqDiffTools [01453d9d-ee7c-5054-8395-0335cb756afa] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SLEEFPirates [476501e8-09a2-5ece-8869-fb82de89a1fa]
      From worker 2:    │   exception = Required dependency VectorizationBase [3d5dd08c-fd9d-11e8-17fa-ed2836048c2f] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SLEEFPirates [476501e8-09a2-5ece-8869-fb82de89a1fa]
      From worker 4:    │   exception = Required dependency VectorizationBase [3d5dd08c-fd9d-11e8-17fa-ed2836048c2f] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    ┌ Warning: Module Distributions with build ID 3326548195295 is missing from the cache.
      From worker 2:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: Module Distributions with build ID 3326595766356 is missing from the cache.
      From worker 4:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    ┌ Warning: Module Distributions with build ID 3326647048939 is missing from the cache.
      From worker 3:    │ This may mean Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] does not support precompilation but is imported by a module that does.
      From worker 3:    └ @ Base loading.jl:1017
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
      From worker 4:    │   exception = Required dependency IniFile [83e8ac13-25f8-5344-8a64-a9f2b223428f] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
      From worker 2:    │   exception = Required dependency IniFile [83e8ac13-25f8-5344-8a64-a9f2b223428f] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597]
      From worker 3:    │   exception = Required dependency StructTypes [856f2bd8-1eba-4b0a-8007-ebc267875bd4] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597]
      From worker 2:    │   exception = Required dependency StructTypes [856f2bd8-1eba-4b0a-8007-ebc267875bd4] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
      From worker 3:    │   exception = Required dependency DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
      From worker 4:    │   exception = Required dependency DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
      From worker 3:    │   exception = Required dependency PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720] failed to load from a cache file.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
      From worker 4:    │   exception = Required dependency PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Neo4j [d2adbeaf-5838-5367-8a2f-e46d570981db]
      From worker 2:    │   exception = Required dependency CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 3:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Neo4j [d2adbeaf-5838-5367-8a2f-e46d570981db]
      From worker 3:    │   exception = Invalid input in module list: expected Logging.
      From worker 3:    └ @ Base loading.jl:1042
      From worker 4:    ┌ Warning: Module LightGraphs with build ID 3381537062764 is missing from the cache.
      From worker 4:    │ This may mean LightGraphs [093fc24a-ae57-5d10-9952-331d41423f4d] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    ┌ Warning: Module LightGraphs with build ID 3381537062764 is missing from the cache.
      From worker 2:    │ This may mean LightGraphs [093fc24a-ae57-5d10-9952-331d41423f4d] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 3:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 2:    │   exception = Required dependency DiffEqDiffTools [01453d9d-ee7c-5054-8395-0335cb756afa] failed to load from a cache file.
      From worker 2:    └ @ Base loading.jl:1042
      From worker 2:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 4:    ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for IncrementalInference [904591bb-b899-562f-9e6f-b8df64c7d480]
      From worker 4:    │   exception = Required dependency LightGraphs [093fc24a-ae57-5d10-9952-331d41423f4d] failed to load from a cache file.
      From worker 4:    └ @ Base loading.jl:1042
      From worker 4:    [ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
      From worker 3:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 3:      ** incremental compilation may be fatally broken for this module **
      From worker 3:
      From worker 2:    ┌ Warning: Module KernelDensityEstimate with build ID 3442860100686 is missing from the cache.
      From worker 2:    │ This may mean KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:1017
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 2:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 2:      ** incremental compilation may be fatally broken for this module **
      From worker 2:
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] failed to load from a cache file.
└ @ Base loading.jl:1042
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 4:    WARNING: Method definition dot(Any, Any, Any) in module LinearAlgebra at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LinearAlgebra/src/generic.jl:924 overwritten in module Optim at /home/leopold/.julia/packages/Optim/Agd3B/src/multivariate/precon.jl:23.
      From worker 4:      ** incremental compilation may be fatally broken for this module **
      From worker 4:
      From worker 4:    ┌ Warning: Module Neo4j with build ID 3387777731462 is missing from the cache.
      From worker 4:    │ This may mean Neo4j [d2adbeaf-5838-5367-8a2f-e46d570981db] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:1017
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
      From worker 4:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
      From worker 2:    [ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: Replacing module `Distributions`
└ @ Base loading.jl:948
┌ Warning: Replacing module `KernelDensityEstimate`
└ @ Base loading.jl:948
      From worker 3:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency KernelDensityEstimate [2472808a-b354-52ea-a80e-1658a3c6056d] failed to load from a cache file.
└ @ Base loading.jl:1042
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
│   exception = Required dependency Neo4j [d2adbeaf-5838-5367-8a2f-e46d570981db] failed to load from a cache file.
└ @ Base loading.jl:1042
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
┌ Warning: Replacing module `Distributions`
└ @ Base loading.jl:948
┌ Warning: Replacing module `KernelDensityEstimate`
└ @ Base loading.jl:948
      From worker 2:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: Replacing module `Distributions`
└ @ Base loading.jl:948
┌ Warning: Replacing module `KernelDensityEstimate`
└ @ Base loading.jl:948
┌ Warning: Replacing module `Neo4j`
└ @ Base loading.jl:948
      From worker 4:    WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME

julia> warmUpSolverJIT()
[ Info: try doautoinit! of x0
STEP 1: Driving around a bit
[ Info: try doautoinit! of x0
[ Info: init with useinitfct [:x0f1]
[ Info: do init of x0
┌ Warning: updateVariableSolverData with solveKey parameter might change in the future, see DFG #565. Future warnings are suppressed
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/services/DFGVariable.jl:526
[ Info: try doautoinit! of x1
[ Info: try doautoinit! of x1
[ Info: init with useinitfct [:x0x1f1]
[ Info: do init of x1
[ Info: try doautoinit! of x2
[ Info: try doautoinit! of x2
[ Info: init with useinitfct [:x1x2f1]
[ Info: do init of x2
[ Info: try doautoinit! of x3
[ Info: try doautoinit! of x3
[ Info: init with useinitfct [:x2x3f1]
[ Info: do init of x3
[ Info: try doautoinit! of x4
[ Info: try doautoinit! of x4
[ Info: init with useinitfct [:x3x4f1]
[ Info: do init of x4
[ Info: try doautoinit! of x5
[ Info: try doautoinit! of x5
[ Info: init with useinitfct [:x4x5f1]
[ Info: do init of x5
[ Info: try doautoinit! of x6
[ Info: try doautoinit! of l1
[ Info: try doautoinit! of x6
[ Info: init with useinitfct [:x5x6f1]
[ Info: do init of x6
[ Info: try doautoinit! of l1
[ Info: init with useinitfct [:x0l1f1, :x6l1f1]
[ Info: do init of l1
[ Info: [2x0p,d2,N100],
[ Info: Ensure variables are all initialized (graphinit)
[ Info: Solving over the Bayes (Junction) tree.
Building Bayes net...
Find potential functions for each clique
[ Info: Bayes Tree Complete
[ Info: Do tree based init-inference on tree
[ Info: Solve is drawing the Bayes tree
Solve Progress: approx max 144, at iter 9        Time: 0:00:05
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:

RemoteException(3, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))RemoteException(4, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:
Stacktrace:error

(RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:error
( [1] error
( [1] error::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; N::Int64N, ::Int64N, ::Int64N, ::Int64dbg, ::Booldbg, ::Booldbg, ::Booldbg, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boollogger, ::Base.CoreLogginglogger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, iters.::SimpleLoggerInt64, iters, ::Int64iters, ::Int64iters, ::Int64drawpdf, ::Booldrawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
) [3]  at __doCliqUpSolveInitialized!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [4] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [5] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333(::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; ::Nothing; ::Nothing; ::Nothing; pollinterval::Float64pollinterval, ::Float64pollinterval, ::Float64pollinterval, ::Float64breakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionverbose, ::Boolverbose, ::Boolverbose, ::Boolverbose, ::Boolverbosefid, ::Baseverbosefid.::TTYBase, verbosefid.::TTYBase, verbosefid.::TTYBase, verboseXtra.::TTYIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, iterlimit.::CliqStatusInt64, iterlimit, ::Int64iterlimit, ::Int64iterlimit, ::Int64injectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::Nothingrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolhousekeeping_cb, ::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencevar."#377#379"{varIncrementalInference"#377#379".{TreeCliquevarIncrementalInference}"#377#379".){TreeClique at varIncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82"#377#379".){TreeClique at 
IncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] .)initStartCliqStateMachine!TreeClique at 
(}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] )initStartCliqStateMachine! at ::(/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::Nothing; ::IncrementalInference.TreeClique, ::Nothing; ::IncrementalInference.TreeClique, oldcliqdata::BayesTreeNodeData::, Nothing; oldcliqdata::BayesTreeNodeData::, Nothing; verbose::Boololdcliqdata, ::BayesTreeNodeDataverbose, ::Boololdcliqdata, ::BayesTreeNodeDataverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baseshow.::TTYBool, drawtree, ::Boolshow, ::Booldrawtree, ::Boolincremental, ::Boolshow, ::Boolincremental, ::Boolshow, ::Boollimititers, ::Int64incremental, ::Boollimititers, ::Int64incremental, ::Boolupsolve, ::Boollimititers, ::Int64upsolve, ::Boollimititers, ::Int64downsolve, ::Boolupsolve, ::Booldownsolve, ::Boolupsolve, ::Boolrecordhistory, ::Booldownsolve, ::Boolrecordhistory, ::Booldownsolve, ::Booldelay, ::Boolrecordhistory, ::Booldelay, ::Boolrecordhistory, ::Boollogger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, logger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, solve_progressbar, ::ProgressMeterlogger.::ProgressUnknownBase.CoreLogging, solve_progressbar.::SimpleLoggerProgressMeter, logger.::ProgressUnknownBase.CoreLogging, algorithm.::SimpleLoggerSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
. [7] ProgressUnknowntryCliqStateMachineSolve!, algorithm(::Symbol
) [7]  at tryCliqStateMachineSolve!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm(::Symbol::)LightDFG at {/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Int64, ::MetaBayesTree, ::Int64, ::MetaBayesTree, ::Nothing; ::Int64, ::Nothing; ::Int64, oldtree::MetaBayesTree::, Nothing; oldtree::MetaBayesTree::, Nothing; verbose::Boololdtree, ::MetaBayesTreeverbose, ::Boololdtree, ::MetaBayesTreeverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baselimititers.::TTYInt64, drawtree, ::Boollimititers, ::Int64drawtree, ::Booldownsolve, ::Boollimititers, ::Int64downsolve, ::Boollimititers, ::Int64incremental, ::Booldownsolve, ::Boolincremental, ::Booldownsolve, ::Booldelaycliqs, ::Arrayincremental{::SymbolBool,delaycliqs, 1::}Array, incremental{::SymbolBool,delaycliqs, 1::}Array, recordcliqs{::SymbolArray,delaycliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, solve_progressbar{1::Symbol}ProgressMeter,, recordcliqs.1::ProgressUnknown}Array, , solve_progressbar{::SymbolProgressMeter,algorithm.1::ProgressUnknown}Symbol, , solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
. [8] ProgressUnknown(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64}), algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(
) [8]  at (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})./task.jl:356(┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
)┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
 at ./task.jl:356┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Error: Task 3 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
[ Info: All cliques should have exited
┌ Error: Task 4 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
Solve Progress: approx max 144, at iter 21       Time: 0:00:09┌ Error: CSM clique 1: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
Solve Progress: approx max 144, at iter 22       Time: 0:00:10[ Info: All cliques should have exited
┌ Error: Task 5 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
┌ Error: CSM clique 2: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
[ Info: All cliques should have exited
┌ Error: Task 6 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
ERROR: TaskFailedException:
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333[ Info: All cliques should have exited
 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:371
 [5] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:280 (repeats 2 times)
 [6] warmUpSolverJIT(; fg::LightDFG{SolverParams,DFGVariable,DFGFactor}, drawtree::Bool) at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:231
 [7] warmUpSolverJIT() at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:226
 [8] top-level scope at REPL[6]:1

julia> [ Info: monitorCSMs: all tasks done

Even running the "preparation steps" from the REPL started in a terminal outside VSCode produced the same errors.

The last thing I gave a shot was using a new project with only RoME added where I tried your "preparation steps". This gave a lot less errors during loading/compilation but still resulted in the error when running warmUpSolverJIT()

leopold@leopold-Desktop:~/Nextcloud/Studium/Masterarbeit$ mkdir julia_sandbox_2
leopold@leopold-Desktop:~/Nextcloud/Studium/Masterarbeit$ cd julia_sandbox_2/
leopold@leopold-Desktop:~/Nextcloud/Studium/Masterarbeit/julia_sandbox_2$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> pwd()
"/home/leopold/Nextcloud/Studium/Masterarbeit/julia_sandbox_2"

(@v1.5) pkg> activate .
 Activating new environment at `~/Nextcloud/Studium/Masterarbeit/julia_sandbox_2/Project.toml`

(julia_sandbox_2) pkg> st
Status `~/Nextcloud/Studium/Masterarbeit/julia_sandbox_2/Project.toml` (empty project)

(julia_sandbox_2) pkg> add RoME
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
  Installed FillArrays ─────── v0.11.1
  Installed SpecialFunctions ─ v1.2.1
  Installed ChainRulesCore ─── v0.9.28
  Installed Distributions ──── v0.24.12
Updating `~/Nextcloud/Studium/Masterarbeit/julia_sandbox_2/Project.toml`
  [91fb55c2] + RoME v0.12.2
Updating `~/Nextcloud/Studium/Masterarbeit/julia_sandbox_2/Manifest.toml`
  [79e6a3ab] + Adapt v3.2.0
  [9bbbb610] + ApproxManifoldProducts v0.1.5
  [ec485272] + ArnoldiMethod v0.1.0
  [4fba245c] + ArrayInterface v2.14.17
  [56f22d72] + Artifacts v1.3.0
  [fbb218c0] + BSON v0.2.6
  [324d7699] + CategoricalArrays v0.8.3
  [d360d2e6] + ChainRulesCore v0.9.28
  [944b1d66] + CodecZlib v0.7.0
  [3da002f7] + ColorTypes v0.10.9
  [5ae59095] + Colors v0.12.6
  [861a8166] + Combinatorics v1.0.2
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.25.0
  [e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
  [150eb455] + CoordinateTransformations v0.6.1
  [9a962f9c] + DataAPI v1.5.1
  [a93c6f00] + DataFrames v0.21.8
  [864edb3b] + DataStructures v0.18.9
  [e2d170a0] + DataValueInterfaces v1.0.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.0.2
  [b4f34e82] + Distances v0.10.2
  [b5cc3c7e] + DistributedFactorGraphs v0.11.3
  [31c24e10] + Distributions v0.24.12
  [ffbed154] + DocStringExtensions v0.8.3
  [e2ba6199] + ExprTools v0.1.3
  [8f5d6c58] + EzXML v1.1.0
  [5789e2e9] + FileIO v1.4.5
  [1a297f60] + FillArrays v0.11.1
  [6a86dc24] + FiniteDiff v2.8.0
  [53c48c17] + FixedPointNumbers v0.8.4
  [f6369f11] + ForwardDiff v0.10.16
  [3e9e306e] + FunctionalStateMachine v0.2.9
  [cd3eb016] + HTTP v0.8.19
  [0e44f5e4] + Hwloc v1.2.0
  [e33a78d0] + Hwloc_jll v2.4.0+0
  [615f187c] + IfElse v0.1.0
  [904591bb] + IncrementalInference v0.20.3
  [d25df0c9] + Inflate v0.1.2
  [83e8ac13] + IniFile v0.5.0
  [41ab1584] + InvertedIndices v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [033835bb] + JLD2 v0.2.4
  [692b3bcd] + JLLWrappers v1.2.0
  [682c06a0] + JSON v0.21.1
  [2535ab7d] + JSON2 v0.3.2
  [2472808a] + KernelDensityEstimate v0.5.4
  [fc18253b] + LazyJSON v0.2.2
  [1fad7336] + LazyStack v0.0.7
  [94ce4f54] + Libiconv_jll v1.16.0+7
  [093fc24a] + LightGraphs v1.3.5
  [d3d80556] + LineSearches v7.1.1
  [1914dd2f] + MacroTools v0.5.6
  [739be429] + MbedTLS v1.0.3
  [c8ffd9c3] + MbedTLS_jll v2.16.8+1
  [626554b9] + MetaGraphs v0.6.6
  [e1d29d7a] + Missings v0.4.5
  [78c3b35d] + Mocking v0.7.1
  [d41bc354] + NLSolversBase v7.7.1
  [2774e3e8] + NLsolve v4.5.1
  [77ba4419] + NaNMath v0.3.5
  [356022a1] + NamedDims v0.2.29
  [d2adbeaf] + Neo4j v2.0.0
  [4d1e1d77] + Nullables v1.0.0
  [6fe1bfb0] + OffsetArrays v1.5.3
  [efe28fd5] + OpenSpecFun_jll v0.5.3+4
  [429524aa] + Optim v1.2.3
  [bac558e1] + OrderedCollections v1.3.3
  [90014a1f] + PDMats v0.10.1
  [d96e819e] + Parameters v0.12.2
  [69de0a69] + Parsers v1.0.15
  [2dfb63ee] + PooledArrays v0.5.3
  [85a6dd25] + PositiveFactorizations v0.2.4
  [92933f4c] + ProgressMeter v1.4.1
  [f8a19df8] + PropertyDicts v0.1.0
  [1fd47b50] + QuadGK v2.4.1
  [3cdcf5f2] + RecipesBase v1.1.1
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v1.1.2
  [79098fc4] + Rmath v0.6.1
  [f50d1b31] + Rmath_jll v0.2.2+1
  [91fb55c2] + RoME v0.12.2
  [6038ab10] + Rotations v1.0.2
  [476501e8] + SLEEFPirates v0.6.8
  [699a6c99] + SimpleTraits v0.9.3
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v1.2.1
  [90137ffa] + StaticArrays v0.12.5
  [2913bbd2] + StatsBase v0.33.2
  [4c63d2b9] + StatsFuns v0.9.6
  [856f2bd8] + StructTypes v1.2.3
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v1.3.2
  [02d47bb6] + TensorCast v0.3.2
  [f269a46b] + TimeZones v1.5.3
  [3bb67fe8] + TranscodingStreams v0.9.5
  [9b8138ad] + TransformUtils v0.2.6
  [3a884ed6] + UnPack v1.0.2
  [cbff2730] + Unmarshal v0.4.2
  [3d5dd08c] + VectorizationBase v0.16.2
  [02c8fc9c] + XML2_jll v2.9.10+3
  [83775a58] + Zlib_jll v1.2.11+18
  [700de1a5] + ZygoteRules v0.2.1
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> using Distributed

julia> using RoME
[ Info: Precompiling RoME [91fb55c2-4c03-5a59-ba21-f4ea956187b8]
[ Info: Multithreaded  convolutions possible, Threads.nthreads()=1.  See `addFactor!(.;threadmodel=MultiThreaded)`.
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
[ Info: KernelDensityEstimate.FORCE_EVAL_DIRECT = true
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
WARNING: ignoring conflicting import of DistributedFactorGraphs.compare into RoME
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactor, use AbstractRelativeRoots instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:318
┌ Warning: BREAKING CHANGE coming to DistributedFactorGraphs v0.12: deprecating AbstractRelativeFactorMinimize, use AbstractRelativeMinimize instead
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/DistributedFactorGraphs.jl:319
[ Info: Including InteractiveUtils related functions in IncrementalInference.

julia> addprocs(3)
3-element Array{Int64,1}:
 2
 3
 4

julia> using RoME

julia> @everywhere using RoME
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).
[ Info: FunctionalStateMachine.jl is adding Graphs.jl related tools (Visualization).

julia> warmUpSolverJIT()
[ Info: try doautoinit! of x0
STEP 1: Driving around a bit
[ Info: try doautoinit! of x0
[ Info: init with useinitfct [:x0f1]
[ Info: do init of x0
┌ Warning: updateVariableSolverData with solveKey parameter might change in the future, see DFG #565. Future warnings are suppressed
└ @ DistributedFactorGraphs ~/.julia/packages/DistributedFactorGraphs/3xqec/src/services/DFGVariable.jl:526
[ Info: try doautoinit! of x1
[ Info: try doautoinit! of x1
[ Info: init with useinitfct [:x0x1f1]
[ Info: do init of x1
[ Info: try doautoinit! of x2
[ Info: try doautoinit! of x2
[ Info: init with useinitfct [:x1x2f1]
[ Info: do init of x2
[ Info: try doautoinit! of x3
[ Info: try doautoinit! of x3
[ Info: init with useinitfct [:x2x3f1]
[ Info: do init of x3
[ Info: try doautoinit! of x4
[ Info: try doautoinit! of x4
[ Info: init with useinitfct [:x3x4f1]
[ Info: do init of x4
[ Info: try doautoinit! of x5
[ Info: try doautoinit! of x5
[ Info: init with useinitfct [:x4x5f1]
[ Info: do init of x5
[ Info: try doautoinit! of x6
[ Info: try doautoinit! of l1
[ Info: try doautoinit! of x6
[ Info: init with useinitfct [:x5x6f1]
[ Info: do init of x6
[ Info: try doautoinit! of l1
[ Info: init with useinitfct [:x0l1f1, :x6l1f1]
[ Info: do init of l1
[ Info: [2x0p,d2,N100],
[ Info: Ensure variables are all initialized (graphinit)
[ Info: Solving over the Bayes (Junction) tree.
Building Bayes net...
Find potential functions for each clique
[ Info: Bayes Tree Complete
[ Info: Do tree based init-inference on tree
[ Info: Solve is drawing the Bayes tree
Solve Progress: approx max 144, at iter 9    Time: 0:00:05
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:

RemoteException(3, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))RemoteException(4, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:
Stacktrace:error

(RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)])) [1] 
Stacktrace:error
( [1] error
( [1] error::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(RemoteException):: at RemoteException./error.jl:42)
 at  [2] ./error.jl:42approxCliqMarginalUp!
( [2] approxCliqMarginalUp!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; ::Array{LikelihoodMessage,1}; N::Int64N, ::Int64N, ::Int64N, ::Int64dbg, ::Booldbg, ::Booldbg, ::Booldbg, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boolmultiproc, ::Boollogger, ::Base.CoreLogginglogger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, logger.::SimpleLoggerBase.CoreLogging, iters.::SimpleLoggerInt64, iters, ::Int64iters, ::Int64iters, ::Int64drawpdf, ::Booldrawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379drawpdf):: at Bool/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
) [3]  at __doCliqUpSolveInitialized!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!
( [3] __doCliqUpSolveInitialized!::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [4] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine
( [4] solveUp_StateMachine::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}):: at CliqStateMachineContainer/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree})
 at  [5] /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333(::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})
( [5] (::StateMachine{CliqStateMachineContainer})::(CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; ::Nothing; ::Nothing; ::Nothing; pollinterval::Float64pollinterval, ::Float64pollinterval, ::Float64pollinterval, ::Float64breakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionbreakafter, ::Functionverbose, ::Boolverbose, ::Boolverbose, ::Boolverbose, ::Boolverbosefid, ::Baseverbosefid.::TTYBase, verbosefid.::TTYBase, verbosefid.::TTYBase, verboseXtra.::TTYIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, verboseXtra.::CliqStatusIncrementalInference, iterlimit.::CliqStatusInt64, iterlimit, ::Int64iterlimit, ::Int64iterlimit, ::Int64injectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::NothinginjectDelayBefore, ::Nothingrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolrecordhistory, ::Boolhousekeeping_cb, ::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencehousekeeping_cb.::IncrementalInferencevar."#377#379"{varIncrementalInference"#377#379".{TreeCliquevarIncrementalInference}"#377#379".){TreeClique at varIncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82"#377#379".){TreeClique at 
IncrementalInference}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] .)initStartCliqStateMachine!TreeClique at 
(}/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82 [6] )initStartCliqStateMachine! at ::(/home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [6] initStartCliqStateMachine!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::MetaBayesTree, ::Nothing; ::IncrementalInference.TreeClique, ::Nothing; ::IncrementalInference.TreeClique, oldcliqdata::BayesTreeNodeData::, Nothing; oldcliqdata::BayesTreeNodeData::, Nothing; verbose::Boololdcliqdata, ::BayesTreeNodeDataverbose, ::Boololdcliqdata, ::BayesTreeNodeDataverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baseshow.::TTYBool, drawtree, ::Boolshow, ::Booldrawtree, ::Boolincremental, ::Boolshow, ::Boolincremental, ::Boolshow, ::Boollimititers, ::Int64incremental, ::Boollimititers, ::Int64incremental, ::Boolupsolve, ::Boollimititers, ::Int64upsolve, ::Boollimititers, ::Int64downsolve, ::Boolupsolve, ::Booldownsolve, ::Boolupsolve, ::Boolrecordhistory, ::Booldownsolve, ::Boolrecordhistory, ::Booldownsolve, ::Booldelay, ::Boolrecordhistory, ::Booldelay, ::Boolrecordhistory, ::Boollogger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, logger, ::Base.CoreLoggingdelay.::SimpleLoggerBool, solve_progressbar, ::ProgressMeterlogger.::ProgressUnknownBase.CoreLogging, solve_progressbar.::SimpleLoggerProgressMeter, logger.::ProgressUnknownBase.CoreLogging, algorithm.::SimpleLoggerSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
. [7] ProgressUnknowntryCliqStateMachineSolve!, algorithm(::Symbol
) [7]  at tryCliqStateMachineSolve!/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63algorithm(::Symbol::)LightDFG at {/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(LightDFG{SolverParams,DFGVariable,DFGFactor}, 
 [7] tryCliqStateMachineSolve!::(MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Int64, ::MetaBayesTree, ::Int64, ::MetaBayesTree, ::Nothing; ::Int64, ::Nothing; ::Int64, oldtree::MetaBayesTree::, Nothing; oldtree::MetaBayesTree::, Nothing; verbose::Boololdtree, ::MetaBayesTreeverbose, ::Boololdtree, ::MetaBayesTreeverbosefid, ::Baseverbose.::TTYBool, verbosefid, ::Baseverbose.::TTYBool, drawtree, ::Boolverbosefid, ::Basedrawtree.::TTYBool, verbosefid, ::Baselimititers.::TTYInt64, drawtree, ::Boollimititers, ::Int64drawtree, ::Booldownsolve, ::Boollimititers, ::Int64downsolve, ::Boollimititers, ::Int64incremental, ::Booldownsolve, ::Boolincremental, ::Booldownsolve, ::Booldelaycliqs, ::Arrayincremental{::SymbolBool,delaycliqs, 1::}Array, incremental{::SymbolBool,delaycliqs, 1::}Array, recordcliqs{::SymbolArray,delaycliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, recordcliqs{1::Symbol}Array,, solve_progressbar{1::Symbol}ProgressMeter,, recordcliqs.1::ProgressUnknown}Array, , solve_progressbar{::SymbolProgressMeter,algorithm.1::ProgressUnknown}Symbol, , solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110algorithm.::ProgressUnknownSymbol, solve_progressbar):: at ProgressMeter/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
. [8] ProgressUnknown(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64}), algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
algorithm(::)Symbol at 
)./task.jl:356 [8]  at ┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})/home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
(
) [8]  at (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})./task.jl:356(┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
)┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
 at ./task.jl:356┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Error: Task 3 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
[ Info: All cliques should have exited
┌ Error: Task 4 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
Solve Progress: approx max 144, at iter 21   Time: 0:00:09┌ Error: CSM clique 1: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
Solve Progress: approx max 144, at iter 22   Time: 0:00:09[ Info: All cliques should have exited
┌ Error: Task 5 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
┌ Error: CSM clique 2: Exit with error state
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:150
[ Info: All cliques should have exited
┌ Error: Task 6 failed, sending error to all cliques
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:183
ERROR: TaskFailedException:
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#433#436"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356

...and 3 more exception(s).

Stacktrace:
 [1] sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333[ Info: All cliques should have exited
 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:371
 [5] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:280 (repeats 2 times)
 [6] warmUpSolverJIT(; fg::LightDFG{SolverParams,DFGVariable,DFGFactor}, drawtree::Bool) at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:231
 [7] warmUpSolverJIT() at /home/leopold/.julia/packages/RoME/W4sOn/src/CanonicalGraphs.jl:226
 [8] top-level scope at REPL[10]:1

julia> [ Info: monitorCSMs: all tasks done

Can you reproduce this error? Instantiating a new environment, adding RoME and doing your prep-steps is a fast thing to check :) I am working on the version registered yesterday by @Affie , but the error is likely to have been introduced earlier (I did not use multiprocess on the up-to-date version for a while). Sorry for the long post ;)

Best, Leo

lemauee commented 3 years ago

tl;dr: Steps to reproduce error Start julia in a an empty folder

(@v1.5) pkg> activate .
(julia_sandbox_2) pkg> add RoME
julia> using Distributed
julia> using RoME
julia> addprocs(3)
julia> using RoME
julia> @everywhere using RoME
julia> warmUpSolverJIT()

Things should crash during solve.

Affie commented 3 years ago

Hi @lemauee, I think there should be an error logged in you "speed" folder from (logpath="speed"). Please check it and post the relevant section. I just saw the post above come through. I'll look in more detail...

lemauee commented 3 years ago

@Affie contents of cliq1_stacktrace.txt:

RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#432#435"{MetaBayesTree,Bool,Bool,Base.TTY,Int64,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64})() at ./threadingconstructs.jl:169
Affie commented 3 years ago

I'm not getting the error on the master branches I'm currently on. I'll try in a fresh environment.

That stack trace just points to the place the error is rethrown. Are there any other error messages?

lemauee commented 3 years ago

I just switched to the master branch in my test-env and still get the same error running dehanns "prep-steps"

Affie commented 3 years ago

It may have to do with one of the dependencies upgradings. I see the original error message is in the first line of your previous post, but it's hard to decipher. It looks like it has something to do with TimeZones.jl or Serialization.jl

lemauee commented 3 years ago

@Affie: The whole log folder and the error message it crashes with of my example speed.zip

[ Info: Bayes Tree Complete
[ Info: Do tree based init-inference on tree
Solve Progress: approx max 24, at iter 4         Time: 0:00:03
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#432#435"{MetaBayesTree,Bool,Bool,Base.TTY,Int64,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64})() at ./threadingconstructs.jl:169┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
┌ Warning: printCliqHistorySummary -- No CSM history found.
└ @ IncrementalInference ~/.julia/packages/IncrementalInference/ShzPg/src/TreeDebugTools.jl:211
ERROR: LoadError: TaskFailedException:
RemoteException(2, CapturedException(KeyError(TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53]), Any[(getindex at dict.jl:467 [inlined], 1), (root_module at loading.jl:968 [inlined], 1), (deserialize_module at Serialization.jl:953, 1), (handle_deserialize at Serialization.jl:855, 1), (deserialize at Serialization.jl:773, 1), (deserialize_datatype at Serialization.jl:1251, 1), (handle_deserialize at Serialization.jl:826, 1), (deserialize at Serialization.jl:773, 1), (handle_deserialize at Serialization.jl:833, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_dict at Serialization.jl:1366, 1), (deserialize at Serialization.jl:1373, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:1350, 1), (handle_deserialize at Serialization.jl:837, 1), (deserialize at Serialization.jl:1338, 1), (handle_deserialize at Serialization.jl:842, 1), (deserialize at Serialization.jl:773, 1), (#5 at Serialization.jl:932, 1), (ntuple at ntuple.jl:18, 1), (deserialize_tuple at Serialization.jl:932, 1), (handle_deserialize at Serialization.jl:816, 1), (deserialize at Serialization.jl:773 [inlined], 1), (deserialize_msg at messages.jl:99, 1), (#invokelatest#1 at essentials.jl:710 [inlined], 1), (invokelatest at essentials.jl:709 [inlined], 1), (message_handler_loop at process_messages.jl:185, 1), (process_tcp_streams at process_messages.jl:142, 1), (#99 at task.jl:356, 1)]))
Stacktrace:
 [1] error(::RemoteException) at ./error.jl:42
 [2] approxCliqMarginalUp!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Array{LikelihoodMessage,1}; N::Int64, dbg::Bool, multiproc::Bool, logger::Base.CoreLogging.SimpleLogger, iters::Int64, drawpdf::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:379
 [3] __doCliqUpSolveInitialized!(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqStateMachineUtils.jl:30
 [4] solveUp_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:333
 [5] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique}) at /home/leopold/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [6] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/CliqueStateMachine.jl:63
 [7] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:110
 [8] (::IncrementalInference.var"#432#435"{MetaBayesTree,Bool,Bool,Base.TTY,Int64,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64})() at ./threadingconstructs.jl:169
Stacktrace:
 [1] [ Info: monitorCSMs: all tasks done
sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/leopold/.julia/packages/IncrementalInference/ShzPg/src/SolverAPI.jl:371
 [5] top-level scope at /home/leopold/Nextcloud/Studium/Masterarbeit/svn/julia/mmiSAM/examples/speed/speed.jl:118
 [6] top-level scope at timing.jl:233
in expression starting at /home/leopold/Nextcloud/Studium/Masterarbeit/svn/julia/mmiSAM/examples/speed/speed.jl:71

julia>
Affie commented 3 years ago

Thanks, it looks like this is the relevant part:

┌ Error: On worker 2:
│ KeyError: key TimeZones [f269a46b-ccf7-5d73-abea-4c690281aa53] not found
│ getindex at ./dict.jl:467 [inlined]
│ root_module at ./loading.jl:968 [inlined]
│ deserialize_module at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:953
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:855
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
│ deserialize_datatype at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1251
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:826
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:833
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1350
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:837
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773 [inlined]
│ deserialize_dict at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1366
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1373
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:842
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1350
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:837
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:1338
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:842
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773
│ #5 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:932
│ ntuple at ./ntuple.jl:18
│ deserialize_tuple at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:932
│ handle_deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:816
│ deserialize at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Serialization/src/Serialization.jl:773 [inlined]
│ deserialize_msg at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/messages.jl:99
│ #invokelatest#1 at ./essentials.jl:710 [inlined]
│ invokelatest at ./essentials.jl:709 [inlined]
│ message_handler_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:185
│ process_tcp_streams at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:142
│ #99 at ./task.jl:356
lemauee commented 3 years ago

TimeZones.jl or Serialization.jl are appearing a lot in the error message, is there any way that this is related to something on my system besides Julia? Its on Ubuntu 18.04. Or if TimeZones is a problem, anything strange connected to my locale and therfore Timezone?

Affie commented 3 years ago

I'm also gettting an error in a fresh environment, but not the same:

caused by: On worker 2:
    TypeError: in new, expected Base.RefValue{Int64}, got a value of type Vector{Float64}
    Stacktrace:
      [1] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1399
      [2] handle_deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:838
      [3] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:774 [inlined]
      [4] deserialize_dict
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1409
      [5] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1416
      [6] handle_deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:843
      [7] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1393
      [8] handle_deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:838
      [9] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1381
     [10] handle_deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:843
     [11] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:774
     [12] #5
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:933
     [13] ntupleany
        @ ./ntuple.jl:43
     [14] deserialize_tuple
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:933
     [15] handle_deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:817
     [16] deserialize
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:774 [inlined]
     [17] deserialize_msg
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/messages.jl:87
     [18] #invokelatest#2
        @ ./essentials.jl:707 [inlined]
     [19] invokelatest
        @ ./essentials.jl:706 [inlined]
     [20] message_handler_loop
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:169
     [21] process_tcp_streams
        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:126
     [22] #99
        @ ./task.jl:406
lemauee commented 3 years ago

Do you still have Julia 1.5.3 lying around? I see youre on 1.6

Affie commented 3 years ago

I'm not getting the error on a fresh environment in julia v1.5

(tmpenv) pkg> st
Status `~/Documents/juliasandbox/tmpenv/Project.toml`
  [91fb55c2] RoME v0.12.2
Affie commented 3 years ago

I can only guess, perhaps try rebuilding TimeZones.jl. Add it to the environment and try ]build TimeZones

lemauee commented 3 years ago

Building TimeZones again did the trick :) , but maybe keeping the issue open due to the 1.6 error is still a good thing.

Should the devs of TimeZones or anybody else maybe know about this problem? Having to rebuild stuff manually does not seem like a good workflow ;)

Affie commented 3 years ago

I'm glad it's working now. Manually building shouldn't be necessary. You can let them know, but it may be hard to reproduce. At least anyone with a similar error from RoME can find this thread.

The julia 1.6-rc1 error is not there in RoME#master, so going to close this. It might have been fixed in the latest IIF.

lemauee commented 3 years ago

Another note on this:

I might have reported this working too early because I had another environment with an old verson of RoME activated when running the "prep-steps". I got the error at some point again. What helped though was activating the right environment on all workers using

@everywhere using Pkg
@everywhere Pkg.activate(".")  # whatever folder your environment files lie in

Maybe this was the source of error all along. Although I don't know If I should have to do this ...

dehann commented 3 years ago

Activating the right environment on all processes is best practice. I'm not 100% sure when Pkg.jl uses the same cached library or separates among processes internally. I have not tested this out too much, but good that we have this data point and time will likely clarify more. we can link back here as we find more. Thanks again for posting!

lemauee commented 3 years ago

As this got me again today, here's a nicer code snippet to activate the same env on all workers:

@everywhere using Pkg
@everywhere Pkg.activate($(Pkg.project().path))
Affie commented 3 years ago

@lemauee for future reference, it looks like you came across this issue: https://github.com/JuliaLang/julia/issues/28781