search

ReactiveBayes / RxInfer.jl

Julia package for automated Bayesian inference on a factor graph with reactive message passing
MIT License
254 stars 24 forks source link

RxInfer causing a segfault/sigterm when called from Evolutionary? #125

Closed John-Boik closed 1 year ago

John-Boik commented 1 year ago

I'm using the genetic algorithm package Evolutionary to help tune/explore some parameters in my RxInfer model. My RxInfer model has rarely caused a segfault/sigterm when run alone, but when called repeatedly by Evolutionary (either Evolutionary.GA or Evolutionary.ES), a sigterm almost always occurs after multiple (about 20) generations. I don't know what is causing the problem, but it does appear to be originating in RxInfer. Can anyone shed light on what the problem might be and/or a possible solution or workaround?

I can't share the code here, but the error message is included below. My formal model is called using the RxInfer.inference() function, the call to which is included in the error message (line 618: fxi at /[path]/RxModel.jl:461, where fxi is my function that contains the call to inference, which occurs on line 461). The first mention of Rocket occurs on line 376. Before that, there are numerous repeating identical or near-identical 15- line sections that start with pthread_cond_wait at.... 

[1070413] signal (15): Terminated
in expression starting at /[path]/RxModel.jl:918
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
#wait#621 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
_trywait at ./asyncevent.jl:138
wait at ./asyncevent.jl:155 [inlined]
profile_printing_listener at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Profile/src/Profile.jl:39
#3 at ./threadingconstructs.jl:373
jfptr_YY.3_59501.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
start_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:1092
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
#wait#621 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
#138 at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:281 [inlined]
lock at ./lock.jl:229
lock at ./condition.jl:78 [inlined]
macro expansion at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:279 [inlined]
#137 at ./threadingconstructs.jl:373
jfptr_YY.137_58196.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
start_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:1092
unknown function (ip: (nil))
pthread_cond_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:883
ijl_task_get_next at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:421
poptask at ./task.jl:974
wait at ./task.jl:983
task_done_hook at ./task.jl:672
jfptr_task_done_hook_31083.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_finish_task at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/task.c:320
jl_threadfun at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/partr.c:132
start_thread at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: (nil))
int64hash at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/support/hashing.c:31
bitmix at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/support/hashing.h:31 [inlined]
typekeyvalue_hash at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jltypes.c:1369 [inlined]
lookup_typevalue at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jltypes.c:891
jl_inst_arg_tuple_type at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jltypes.c:1807
arg_type_tuple at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2100 [inlined]
jl_lookup_generic_ at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2884 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2936
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:255 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:203 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:101 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:68 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb54826)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202
unknown function (ip: 0x7f6cddb54586)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:97
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:87 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cd080c806)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:255 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:203 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:101 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:68 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cd080b636)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202
unknown function (ip: 0x7f6cddb3aec6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:97
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:87 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb846b6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:255 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:203 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:101 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:68 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb924c6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202
unknown function (ip: 0x7f6cddb54586)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb872a2)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:207 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb87202)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:207 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cd08163c6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/tap.jl:63 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cd08162a6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:255 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:203 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:101 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:68 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cd08150b6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202
unknown function (ip: 0x7f6cddb3aec6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:97
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/collected.jl:87 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb95f66)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
#7 at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
limitstack at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:52
scheduled_next! at /home/jboik/.julia/packages/RxInfer/12xMs/src/rocket.jl:86 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:255 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:203 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/scheduled.jl:12 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:101 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined.jl:68 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
next_received! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:65 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/observable/combined_updates.jl:34 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:206 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/operators/map.jl:62 [inlined]
scheduled_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/schedulers/asap.jl:23
unknown function (ip: 0x7f6cddb95cf6)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/subject.jl:62
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/subjects/recent.jl:62 [inlined]
actor_on_next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:250 [inlined]
next! at /home/jboik/.julia/packages/Rocket/EGm1W/src/actor.jl:202 [inlined]
update! at /home/jboik/.julia/packages/ReactiveMP/28SY0/src/variables/data.jl:143 [inlined]
update! at /home/jboik/.julia/packages/ReactiveMP/28SY0/src/variables/data.jl:119 [inlined]
#220 at /home/jboik/.julia/packages/ReactiveMP/28SY0/src/variables/data.jl:152 [inlined]
foreach at ./abstractarray.jl:3073
update! at /home/jboik/.julia/packages/ReactiveMP/28SY0/src/variables/data.jl:151
unknown function (ip: 0x7f6cddb90c66)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
#inference#156 at /home/jboik/.julia/packages/RxInfer/12xMs/src/inference.jl:652
inference at /home/jboik/.julia/packages/RxInfer/12xMs/src/inference.jl:478
unknown function (ip: 0x7f6cd07dedc2)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
fxi at /[path]/RxModel.jl:461
unknown function (ip: 0x7f6cd07d0961)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
fxi2 at /[path]/RxModel.jl:347
#182 at /[path]/RxModel.jl:390 [inlined]
value at /home/jboik/.julia/packages/Evolutionary/65hL6/src/api/objective.jl:52 [inlined]
#91 at /home/jboik/.julia/packages/Evolutionary/65hL6/src/es.jl:68 [inlined]
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:840 [inlined]
collect_to_with_first! at ./array.jl:818 [inlined]
_collect at ./array.jl:812
collect_similar at ./array.jl:711 [inlined]
map at ./abstractarray.jl:3261 [inlined]
initial_state at /home/jboik/.julia/packages/Evolutionary/65hL6/src/es.jl:68
optimize at /home/jboik/.julia/packages/Evolutionary/65hL6/src/api/optimize.jl:70
unknown function (ip: 0x7f6cd0822b86)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
optimize at /home/jboik/.julia/packages/Evolutionary/65hL6/src/api/optimize.jl:55
optimize at /home/jboik/.julia/packages/Evolutionary/65hL6/src/api/optimize.jl:42
unknown function (ip: 0x7f6cd07c8c36)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
main at /[path]/RxModel.jl:389
unknown function (ip: 0x7f6cd07c075f)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1864
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
_include at ./loading.jl:1924
include at ./client.jl:478
unknown function (ip: 0x7f6e2402a3e2)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:856
eval_body at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:568
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
eval_user_input at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
repl_backend_loop at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
#start_repl_backend#46 at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
start_repl_backend at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:231
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
#run_repl#59 at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
run_repl at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
jfptr_run_repl_60276.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
#1017 at ./client.jl:421
jfptr_YY.1017_32248.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:816 [inlined]
invokelatest at ./essentials.jl:813 [inlined]
run_main_repl at ./client.jl:405
exec_options at ./client.jl:322
_start at ./client.jl:522
jfptr__start_37386.clone_1 at /home/jboik/usr/julia-1.9.0-linux-x86_64/julia-1.9.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
true_main at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:573
jl_repl_entrypoint at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:717
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
unknown function (ip: (nil))
Allocations: 30869460694 (Pool: 30867859530; Big: 1601164); GC: 5630
Terminated
bvdmitri commented 1 year ago

Thats weird...Do you use the options = (limit_stack_depth = 500, ) option for the inference function? Could you try to run inference without this option?, it is not always necessary (only for large models). This is the only place where we use Tasks to emulate infinite stack-trace in our code and it may causing this issue, but I would say it's a bug in Julia or a buggy interaction between RxInfer and Evolutionary. I don't know how Evolutionary works.

bvdmitri commented 1 year ago

I can see in the stack trace provided that the segfault happens in limitstack function so it is definitely related to the options = (limit_stack_depth = 500, ) setting. I guess Evolutionary tries to run several inference procedures in parallel and this may cause problems with async Tasks?

John-Boik commented 1 year ago

I can also verify that the sigterm fault arises when using Optim minimization (e.g., LBFGS), rather than Evolutionary. It occurs after 700 to 1700 function calls. Good to know that the fault is related to limit_stack_depth. My problem is large, however, and will not run without limit_stack_depth. I use limit_stack_depth=900. Any ideas on preventing the sigterm fault for this situation?

Also, the sigterm happens whether I run Evolutionary using multiple threads (parallelization = :thread) or not.

bvdmitri commented 1 year ago

I'm not sure, we've never encountered this issue before and generally segfaults should never occur in Julia. We do not do anything "forbidden" in Julia and only the standard Task object. I would say its a bug in the Task object. Would it be too difficult to verify if an issue also occurs in different versions of Julia and also on the master branch? Tthat would require you to compile the Julia itself, but the process is quite easy, you basically just need to clone the Julia repository and run make.

John-Boik commented 1 year ago

Looks like it was a memory issue on my end, and a non-graceful exit on the part of Linux. I updated Ubuntu from 20.04 to 23.04 and now the exit is graceful (a clean kill event). I also fixed the memory issue.

As a late addendum after I closed this issue, I have a perhaps helpful tip. In my code, an outer optimization function (e.g., LBFGS) repeatedly calls a subfunction that contains result = RxInfer.inference(...). In order to prevent memory consumption from blowing up (at about .1GB per 10 seconds), I needed to include the following prior to the return of the subfunction:

result = nothing
GC.gc()

The above solved my memory issue.