search

ICB-DCM / pyABC

distributed, likelihood-free inference
https://pyabc.rtfd.io
BSD 3-Clause "New" or "Revised" License
204 stars 44 forks source link

Problem using multiple cores with pyABC and PyJulia #595

Closed freddie090 closed 10 months ago

freddie090 commented 1 year ago

I am getting an error when trying to run pyABC which calls an ODE model in Julia. The issue only arises when:

Here is the error message: 

>>> h = abc.run(max_nr_populations=5)

ABC INFO: Calibration sample t = -1.
julia: /buildworker/worker/package_linux64/build/src/debuginfo.cpp:1602: void register_eh_frames(uint8_t*, size_t): Assertion `end_ip != 0' failed.
signal (6): Aborted
in expression starting at none:4
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base.cold.0 at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
register_eh_frames at /buildworker/worker/package_linux64/build/src/debuginfo.cpp:1602
finalizeMemory at /buildworker/worker/package_linux64/build/src/cgmemmgr.cpp:894
_ZZN4llvm15RuntimeDyldImpl13finalizeAsyncESt10unique_ptrIS0_St14default_deleteIS0_EENS_15unique_functionIFvNS_6object12OwningBinaryINS6_10ObjectFileEEES1_INS_11RuntimeDyld16LoadedObjectInfoES2_ISB_EENS_5ErrorEEEES9_SD_ENUlNS_8ExpectedISt3mapINS_9StringRefENS_18JITEvaluatedSymbolESt4lessISJ_ESaISt4pairIKSJ_SK_EEEEEE_clESS_ at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJNS_8ExpectedISt3mapINS_9StringRefENS_18JITEvaluatedSymbolESt4lessIS4_ESaISt4pairIKS4_S5_EEEEEEE8CallImplIZNS_15RuntimeDyldImpl13finalizeAsyncESt10unique_ptrISG_St14default_deleteISG_EENS_15unique_functionIFvNS_6object12OwningBinaryINSM_10ObjectFileEEESH_INS_11RuntimeDyld16LoadedObjectInfoESI_ISR_EENS_5ErrorEEEESP_ST_EUlSD_E_EEvPvRSD_ at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJNS_8ExpectedINS_8DenseMapINS_3orc15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoIS5_EENS0_12DenseMapPairIS5_S6_EEEEEEEE8CallImplIZN12_GLOBAL__N_127JITDylibSearchOrderResolver6lookupERKSt3setINS_9StringRefESt4lessISI_ESaISI_EENS_15unique_functionIFvNS2_ISt3mapISI_S6_SK_SaISt4pairIKSI_S6_EEEEEEEEEUlSC_E_EEvPvRSC_ at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc23AsynchronousSymbolQuery14handleCompleteEv at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_EEEENSG_ISD_EENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_EENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_EENSN_IS6_SV_EEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN12_GLOBAL__N_127JITDylibSearchOrderResolver6lookupERKSt3setIN4llvm9StringRefESt4lessIS3_ESaIS3_EENS2_15unique_functionIFvNS2_8ExpectedISt3mapIS3_NS2_18JITEvaluatedSymbolES5_SaISt4pairIKS3_SD_EEEEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm15RuntimeDyldImpl13finalizeAsyncESt10unique_ptrIS0_St14default_deleteIS0_EENS_15unique_functionIFvNS_6object12OwningBinaryINS6_10ObjectFileEEES1_INS_11RuntimeDyld16LoadedObjectInfoES2_ISB_EENS_5ErrorEEEES9_SD_ at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm13jitLinkForORCENS_6object12OwningBinaryINS0_10ObjectFileEEERNS_11RuntimeDyld13MemoryManagerERNS_17JITSymbolResolverEbNS_15unique_functionIFNS_5ErrorERKS2_RNS4_16LoadedObjectInfoESt3mapINS_9StringRefENS_18JITEvaluatedSymbolESt4lessISG_ESaISt4pairIKSG_SH_EEEEEENS9_IFvS3_St10unique_ptrISD_St14default_deleteISD_EESA_EEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc24RTDyldObjectLinkingLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EES2_INS_12MemoryBufferES4_IS7_EE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession26materializeOnCurrentThreadESt10unique_ptrINS0_19MaterializationUnitESt14default_deleteIS3_EES2_INS0_29MaterializationResponsibilityES4_IS7_EE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZNSt17_Function_handlerIFvSt10unique_ptrIN4llvm3orc19MaterializationUnitESt14default_deleteIS3_EES0_INS2_29MaterializationResponsibilityES4_IS7_EEEPSA_E9_M_invokeERKSt9_Any_dataOS6_OS9_ at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_EEEENSG_ISD_EENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_EENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_EENSN_IS6_SV_EEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_EEEENSL_IS5_EENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /home/fwhiting/julia-1.7.2/bin/../lib/julia/libLLVM-12jl.so (unknown line)
addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:779
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1059
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1103
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1125 [inlined]
_jl_compile_codeinst at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:154
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:350
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1980
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:2246 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2239 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
convert at /home/fwhiting/.julia/packages/PyCall/twYvK/src/conversions.jl:835
julia_args at /home/fwhiting/.julia/packages/PyCall/twYvK/src/callback.jl:18 [inlined]
_pyjlwrap_call at /home/fwhiting/.julia/packages/PyCall/twYvK/src/callback.jl:24
unknown function (ip: 0x7ff42439d9e5)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
pyjlwrap_call at /home/fwhiting/.julia/packages/PyCall/twYvK/src/callback.jl:44
unknown function (ip: 0x7ff42437f8e2)
_PyObject_MakeTpCall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_FastCallDict at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_Call_Prepend at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
slot_tp_call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_MakeTpCall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyVectorcall_Call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
method_vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_FastCallDict at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_Call_Prepend at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
slot_tp_init at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
type_call.lto_priv.242 at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyObject_MakeTpCall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyVectorcall_Call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
method_vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
method_vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
method_vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyVectorcall_Call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
method_vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyEval_EvalCodeEx at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyEval_EvalCode at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
builtin_exec.lto_priv.1736 at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
cfunction_vectorcall_FASTCALL at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
function_code_fastcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
call_function at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyVectorcall_Call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
_PyFunction_Vectorcall at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
PyVectorcall_Call at /opt/software/applications/anaconda/3/lib/libpython3.8.so.1.0 (unknown line)
macro expansion at /home/fwhiting/.julia/packages/PyCall/twYvK/src/exception.jl:108 [inlined]
#107 at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:43 [inlined]
disable_sigint at ./c.jl:458 [inlined]
__pycall! at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:42 [inlined]
_pycall! at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:29
_pycall! at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:11 [inlined]
#_#114 at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:86 [inlined]
PyObject at /home/fwhiting/.julia/packages/PyCall/twYvK/src/pyfncall.jl:86
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:126
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:215
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:166 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:587
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:731
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:885
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:830
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
exec_options at ./client.jl:268
_start at ./client.jl:495
jfptr__start_38732.clone_1 at /home/fwhiting/julia-1.7.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:559
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:701
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x400808)
Allocations: 57339906 (Pool: 57328471; Big: 11435); GC: 26
ABC.History INFO: Done <ABCSMC id=1, duration=0:00:10.414569, end_time=2023-01-27 04:02:08>
Traceback (most recent call last):
  File "/data/scratch/DMP/UCEC/GENEVOD/fwhiting/barcode_project/python_venvs/py_abc_julia/lib/python3.8/site-packages/pyabc/sampler/multicorebase.py", line 99, in get_if_worker_healthy
    item = queue.get(True, 5)
  File "/opt/software/applications/anaconda/3/lib/python3.8/multiprocessing/queues.py", line 108, in get
    raise Empty
_queue.Empty

Here is the julia model: 

module ODE_Test

# Testing an ODE model for ABC to check why failing on the cluster. 

using DifferentialEquations

# Function that calls a simple exponential growth model ODE. 

function ode_exp_growth(lam)

    p = [lam]
    u0 = [1]
    tspan = (0.0, 6.0)

    function ode_grow!(du, u, p, t)

        n = u[1]
        lam = p[1]

        du[1] = dn = lam*n

    end

    prob = ODEProblem(ode_grow!, u0,tspan,p)

    sol = solve(prob, saveat = 0.1)

    return sol

end

function model(par)

    lam = par["lam"]
    sol = ode_exp_growth(lam)
    return Dict("t"=>sol.t, "u"=>sol.u)

end

# Ground truth 
lam = 0.693

# Observed data
observation = model(Dict("lam"=>lam))

# Distance function
function distance(y, y0)
    u, u0 = y["u"], y0["u"]
    if length(u) != length(u0)
        throw(AssertionError("Dimension mismatch"))
    end
    return sum((u .- u0).^2) / length(u0)
end

end  # module

and here is the python script used to call it: 

import os 
import tempfile

import matplotlib.pyplot as plt
import numpy as np

import pyabc
from pyabc import ABCSMC, RV, Distribution, MulticoreEvalParallelSampler, SingleCoreSampler
from pyabc.external.julia import Julia

pyabc.settings.set_figure_params('pyabc')  # for beautified plots

jl = Julia(module_name="ODE_Test", 
           source_file="abc_models/ODE_Test.jl")

model = jl.model()
distance = jl.distance()
obs = jl.observation()

gt_par = {"lam": 0.693}

par_limits = {
    "lam": (0, 1)
}

prior = Distribution(
    **{key: RV("uniform", lb, ub - lb) for key, (lb, ub) in par_limits.items()}
)

abc = ABCSMC(
    model,
    prior,
    distance,
    sampler=MulticoreEvalParallelSampler(n_procs=1)
)

db = tempfile.mkstemp(suffix=".db")[1]
abc.new("sqlite:///" + db, obs)
h = abc.run(max_nr_populations=5)
yannikschaelte commented 1 year ago

First idea since it only affects multicore: Sometimes Julia has pre-compilation issues. Maybe you could once run all components (on float-valued parameters/data) in the main thread, as shown here https://pyabc.readthedocs.io/en/latest/examples/using_julia.html Cell 6.

freddie090 commented 1 year ago

Hi - thanks for the quick reply. I can run model(gt_par) from python and the model runs fine. It is only when it reaches abc.run that the problem arises. Is this what you meant?

yannikschaelte commented 1 year ago

Hi, sorry for the delay. Yes, indeed that is what I meant - this should allow all necessary pre-compilation to happen, but need to look more into the Julia side. I hope that for the moment, single-core execution suffices. I will try to reproduce the problem.

freddie090 commented 1 year ago

Hey, yes I've managed with single-core for now.

Just looking around the only things I could find that might point to the problem are here: https://discourse.julialang.org/t/abc-inference-with-parallel-true-gives-error-in-differentialequations-jl/50681 https://github.com/JuliaPy/PyCall.jl/issues/373

I'm assuming something in the ODE solver in Julia is the issue as it runs fine with the discrete jump process version of the sim on multi-core.

stephanmg commented 1 year ago

Hello @freddie090, is this now an issue with Julia or pyABC? If on the Julia side, I would close the pyABC issue here. Let me know.

stephanmg commented 10 months ago

Closing due to inactivity over last 6 months. Please re-open issue if required.