search

PandABlocks / PandABlocks-FPGA

VHDL functional blocks with their simulations and test sequences
Apache License 2.0
19 stars 16 forks source link

hdl_tests (Vivado sims) seg faulting in Vivado2022.2 #119

Closed glennchid closed 1 year ago

glennchid commented 1 year ago

Tests are seg faulting under both RHEL7 and RHEL8 at what seems to be random parts of the simulations

glennchid commented 1 year ago

Stacktrace:

#
# An unexpected error has occurred (11)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7fdd0a913b80]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb8f8f4) [0x7fdd0bdb48f4]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb946ab) [0x7fdd0bdb96ab]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fdd0633c56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7fdd0637f945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7fdd063870f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7fdd0633e216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x25a57a) [0x7fdcf225157a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fdd0633c56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x34bf8) [0x7fdd0633dbf8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fdd0633e163]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7fdd063a2d2a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2752c0) [0x7fdcf226c2c0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fdd0bdb930f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fdd0633c56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fdd0633c6a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fdd0633e4c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bb7f8) [0x7fdcf22b27f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7fdcf22b3dff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fdd0bdb930f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fdd0633c56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fdd0633c6a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fdd0633e4c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7fdd0ca5f3d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7fdd063a92f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7fdd0bde67ab]
/lib64/libpthread.so.0(+0x81ca) [0x7fdd0a6ad1ca]
/lib64/libc.so.6(clone+0x43) [0x7fdd0a8fee73]
EmilioPeJu commented 1 year ago

I checked the offset 0x4eb80 inside libc and found the following:

(gdb) disass 0x4eb80
Dump of assembler code for function __restore_rt:
   0x000000000004eb80 <+0>: mov    rax,0xf
   0x000000000004eb87 <+7>: syscall 
   0x000000000004eb89 <+9>: nop    DWORD PTR [rax+0x0]
End of assembler dump.

We can see that it's using syscall 15 which correspond to rt_sigreturn... all this suggests it's a problem returning from a signal handler

EDIT: now that I think about it, the signal is the consequence of a segfault

glennchid commented 1 year ago

Triggered from Vivado GUI:

vivado: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
Abnormal program termination (6)
Please check '/scratch/clm61942/PandA/PandABlocks-FPGA/build/targets/PandABox/tests/regression_tests/hs_err_pid2720374.log' for details

contents of hs_err_pid2720374.log

#
# An unexpected error has occurred (6)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7f14affc2b80]
/lib64/libc.so.6(gsignal+0x10f) [0x7f14affc2aff]
/lib64/libc.so.6(abort+0x127) [0x7f14aff95ea5]
/lib64/libc.so.6(+0x21d79) [0x7f14aff95d79]
/lib64/libc.so.6(+0x47456) [0x7f14affbb456]
/lib64/libpthread.so.0(__pthread_mutex_lock+0x241) [0x7f14afd5ecc1]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(+0x1a538) [0x7f14ae640538]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(boost::thread::native_handle()+0x26) [0x7f14ae63a1d6]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbSharedMemReader::restart()+0x1f) [0x7f146ae61bff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::DataReader::restart()+0xe4) [0x7f1474b83114]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::FrontEndControl::runRestart() const+0x5d) [0x7f1474b19d4d]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObjectImp::issueBlockingCommand(RSDData::CommandContainer const&, HDGUIStatus*, Tcl_Interp*)+0x883) [0x7f147cb8a393]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::issueRunRestart()+0x36) [0x7f147cb8a626]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simtcltasks.so(+0x21c4c8) [0x7f1429ffb4c8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f14b146830f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f14ab9eb56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7f14aba2e945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7f14aba360f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7f14ab9ed216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb91c33) [0x7f14b1465c33]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_ServiceEvent+0x7f) [0x7f14aba5fbef]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_DoOneEvent+0x154) [0x7f14aba5ff24]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2b3c27) [0x7f1497959c27]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bc4f7) [0x7f14979624f7]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7f1497962dff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f14b146830f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f14ab9eb56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7f14ab9eb6a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7f14ab9ed4c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7f14b210e3d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7f14aba582f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7f14b14957ab]
/lib64/libpthread.so.0(+0x81ca) [0x7f14afd5c1ca]
/lib64/libc.so.6(clone+0x43) [0x7f14affade73]
glennchid commented 1 year ago

Also triggered from Vivado GUI:

****** Vivado v2022.2 (64-bit)
  **** SW Build 3671981 on Fri Oct 14 04:59:54 MDT 2022
  **** IP Build 3669848 on Fri Oct 14 08:30:02 MDT 2022
    ** Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.

start_gui
terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
Abnormal program termination (6)
Please check '/scratch/clm61942/PandA/PandABlocks-FPGA/build/targets/PandABox/tests/regression_tests/hs_err_pid2715927.log' for details

[2]+  Exit 134                vivado regression_tests.xpr

Contents of hs_err_pid2715927.log: 

#
# An unexpected error has occurred (6)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7fe492a3cb80]
/lib64/libc.so.6(gsignal+0x10f) [0x7fe492a3caff]
/lib64/libc.so.6(abort+0x127) [0x7fe492a0fea5]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libstdc++.so.6(+0x8ba13) [0x7fe493056a13]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libstdc++.so.6(+0x91b86) [0x7fe49305cb86]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libstdc++.so.6(+0x90c79) [0x7fe49305bc79]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libstdc++.so.6(__gxx_personality_v0+0x2c5) [0x7fe49305c5c5]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libgcc_s.so.1(+0x104e3) [0x7fe492dc44e3]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/Rhel/8/libgcc_s.so.1(_Unwind_Resume+0x125) [0x7fe492dc4d65]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(+0x11665) [0x7fe4910b1665]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbSharedMemReader::restart()+0x44) [0x7fe44d8dbc24]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::DataReader::restart()+0xe4) [0x7fe4575fd114]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::FrontEndControl::runRestart() const+0x5d) [0x7fe457593d4d]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObjectImp::issueBlockingCommand(RSDData::CommandContainer const&, HDGUIStatus*, Tcl_Interp*)+0x883) [0x7fe45f604393]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::issueRunRestart()+0x36) [0x7fe45f604626]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simtcltasks.so(+0x21c4c8) [0x7fe40f5bd4c8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fe493ee230f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fe48e46556f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7fe48e4a8945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7fe48e4b00f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7fe48e467216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb91c33) [0x7fe493edfc33]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_ServiceEvent+0x7f) [0x7fe48e4d9bef]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_DoOneEvent+0x154) [0x7fe48e4d9f24]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2b3c27) [0x7fe47a3d3c27]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bc4f7) [0x7fe47a3dc4f7]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7fe47a3dcdff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fe493ee230f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fe48e46556f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fe48e4656a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fe48e4674c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7fe494b883d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7fe48e4d22f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7fe493f0f7ab]
/lib64/libpthread.so.0(+0x81ca) [0x7fe4927d61ca]
/lib64/libc.so.6(clone+0x43) [0x7fe492a27e73]
glennchid commented 1 year ago

Similar to original case, from scripted run:

Simulation has finished
$finish called at time : 1092 ns : File "/scratch/clm61942/PandA/PandABlocks-FPGA/build/hdl_timing/clock/timing001/hdl_timing.v" Line 130
Memory in use : 121420 KB (peak memory: 178952 KB)     CPU usage : 0 ms
Abnormal program termination (11)
Please check '/scratch/clm61942/PandA/PandABlocks-FPGA/build/targets/PandABox/tests/hs_err_pid2731454.log' for details
segfault in /dls_sw/FPGA/Xilinx/Vivado/2022.2/bin/unwrapped/lnx64.o/vivado -exec vivado -mode batch -notrace -source /scratch/clm61942/PandA/PandABlocks-FPGA/tests/hdl/regression_tests.tcl -tclargs /scratch/clm61942/PandA/PandABlocks-FPGA /scratch/clm61942/PandA/PandABlocks-FPGA/targets/PandABox /scratch/clm61942/PandA/PandABlocks-FPGA/build/targets/PandABox /scratch/clm61942/PandA/PandABlocks-FPGA/build /scratch/clm61942/PandA/PandABlocks-FPGA/build/apps/PandABox-no-fmc bits calc clock counter div filter lut pcap pcomp pgen posenc pulse qdec seq srgate, exiting...
make: *** [Makefile:209: hdl_test] Error 139

Content of hs_err_pid2731454.log: 

#
# An unexpected error has occurred (11)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7feb30be5b80]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb8f8f4) [0x7feb320868f4]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb946ab) [0x7feb3208b6ab]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7feb2c60e56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7feb2c651945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7feb2c6590f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7feb2c610216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x25a57a) [0x7feb1852357a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7feb2c60e56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x34bf8) [0x7feb2c60fbf8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7feb2c610163]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7feb2c674d2a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2752c0) [0x7feb1853e2c0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7feb3208b30f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7feb2c60e56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7feb2c60e6a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7feb2c6104c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bb7f8) [0x7feb185847f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7feb18585dff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7feb3208b30f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7feb2c60e56f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7feb2c60e6a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7feb2c6104c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7feb32d313d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7feb2c67b2f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7feb320b87ab]
/lib64/libpthread.so.0(+0x81ca) [0x7feb3097f1ca]
/lib64/libc.so.6(clone+0x43) [0x7feb30bd0e73]
EmilioPeJu commented 1 year ago

One of the ways it fails seems related to the following redhat ticket: https://bugzilla.redhat.com/show_bug.cgi?id=570278 It suggest a mutex is being double unlocked, or used without being initialised

glennchid commented 1 year ago

From RockyLinux 8.5 container:

The test result is seq_12_tb
##################################### seq_12_tb has passed #####################################
vivado: ../nptl/pthread_mutex_lock.c:433: __pthread_mutex_lock_full: Assertion `INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust' failed.
Abnormal program termination (6)
Please check '/root/repo/PandABlocks-FPGA/build/targets/PandABox/tests/hs_err_pid781.log' for details
make: *** [Makefile:209: hdl_test] Error 134
sh-4.4# cat /etc/redhat-release 
Rocky Linux release 8.5 (Green Obsidian)

Stack-trace:

sh-4.4# cat build/targets/PandABox/tests/hs_err_pid781.log 
#
# An unexpected error has occurred (6)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7f301af3fb80]
/lib64/libc.so.6(gsignal+0x10f) [0x7f301af3faff]
/lib64/libc.so.6(abort+0x127) [0x7f301af12ea5]
/lib64/libc.so.6(+0x21d79) [0x7f301af12d79]
/lib64/libc.so.6(+0x47456) [0x7f301af38456]
/lib64/libpthread.so.0(+0xaa76) [0x7f301acdba76]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(+0x1a538) [0x7f30195bd538]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(boost::thread::native_handle()+0x26) [0x7f30195b71d6]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbSharedMemReader::~WdbSharedMemReader()+0x1b) [0x7f2fe35b0d0b]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbLiveReader::~WdbLiveReader()+0x53) [0x7f2fe359f863]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbLiveReader::~WdbLiveReader()+0x9) [0x7f2fe359fb19]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::DataObjMgr::~DataObjMgr()+0x39) [0x7f2fed305799]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(+0x2a6442) [0x7f2fed327442]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::DataReader::~DataReader()+0x28) [0x7f2fed3169c8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(+0x2ebaea) [0x7f2fed36caea]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::LiveSimClient::~LiveSimClient()+0x9) [0x7f2fed36b409]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::closeLiveSimClient()+0x6b) [0x7f2ff531885b]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::~RSDLiveSimulationObject()+0x2c) [0x7f2ff5318a1c]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::~RSDLiveSimulationObject()+0x9) [0x7f2ff5318a59]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simtcltasks.so(+0x1ff5fa) [0x7f2fd9f275fa]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f301c3e530f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f301696856f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7f30169ab945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7f30169b30f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7f301696a216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x25a57a) [0x7f301009b57a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f301696856f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x34bf8) [0x7f3016969bf8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7f301696a163]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7f30169ced2a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2752c0) [0x7f30100b62c0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f301c3e530f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f301696856f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7f30169686a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7f301696a4c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bb7f8) [0x7f30100fc7f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7f30100fddff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f301c3e530f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f301696856f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7f30169686a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7f301696a4c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7f301d08b3d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7f30169d52f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7f301c4127ab]
/lib64/libpthread.so.0(+0x81cf) [0x7f301acd91cf]
/lib64/libc.so.6(clone+0x43) [0x7f301af2ae73]
sh-4.4#
glennchid commented 1 year ago

Also, from Rocky 8.5

Abnormal program termination (11)
Please check '/root/repo/PandABlocks-FPGA/build/targets/PandABox/tests/hs_err_pid1188.log' for details
segfault in /dls_sw/FPGA/Xilinx/Vivado/2022.2/bin/unwrapped/lnx64.o/vivado -exec vivado -mode batch -notrace -source /root/repo/PandABlocks-FPGA/tests/hdl/regression_tests.tcl -tclargs /root/repo/PandABlocks-FPGA /root/repo/PandABlocks-FPGA/targets/PandABox /root/repo/PandABlocks-FPGA/build/targets/PandABox /root/repo/PandABlocks-FPGA/build /root/repo/PandABlocks-FPGA/build/apps/PandABox-no-fmc bits calc clock counter div filter lut pcap pcomp pgen posenc pulse qdec seq srgate, exiting...
make: *** [Makefile:209: hdl_test] Error 139
sh-4.4# cat build/targets/PandABox/tests/hs_err_pid1188.log 
#
# An unexpected error has occurred (11)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7f6f258cab80]
/lib64/libpthread.so.0(pthread_detach+0x4) [0x7f6f25665474]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libboost_thread.so(boost::thread::detach()+0x61) [0x7f6f23f42071]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simbridge_kernel.so(SimBridge::WdbSharedMemReader::restart()+0x44) [0x7f6eedf3bc24]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::DataReader::restart()+0xe4) [0x7f6ef7c9f114]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::FrontEndControl::runRestart() const+0x5d) [0x7f6ef7c35d4d]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObjectImp::issueBlockingCommand(RSDData::CommandContainer const&, HDGUIStatus*, Tcl_Interp*)+0x883) [0x7f6effca6393]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::issueRunRestart()+0x36) [0x7f6effca6626]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simtcltasks.so(+0x21c4c8) [0x7f6ee48cf4c8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f6f26d7030f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f6f212f356f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x76945) [0x7f6f21336945]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x7e0f9) [0x7f6f2133e0f9]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7f6f212f5216]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x25a57a) [0x7f6f1aa2657a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f6f212f356f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x34bf8) [0x7f6f212f4bf8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7f6f212f5163]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7f6f21359d2a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2752c0) [0x7f6f1aa412c0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f6f26d7030f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f6f212f356f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7f6f212f36a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7f6f212f54c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bb7f8) [0x7f6f1aa877f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7f6f1aa88dff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7f6f26d7030f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7f6f212f356f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7f6f212f36a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7f6f212f54c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7f6f27a163d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7f6f213602f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7f6f26d9d7ab]
/lib64/libpthread.so.0(+0x81cf) [0x7f6f256641cf]
/lib64/libc.so.6(clone+0x43) [0x7f6f258b5e73]
sh-4.4#
glennchid commented 1 year ago

New error, under RHEL8, with uprobe on librdi_common:

launch_simulation: Time (s): cpu = 00:00:04 ; elapsed = 00:00:06 . Memory (MB): peak = 2043.660 ; gain = 98.902 ; free physical = 4851 ; free virtual = 20486
Abnormal program termination (11)
Please check '/scratch/clm61942/PandA/PandABlocks-FPGA/standalone/hs_err_pid3310794.log' for details
segfault in /dls_sw/FPGA/Xilinx/Vivado/2022.2/bin/unwrapped/lnx64.o/vivado -exec vivado -mode batch -source standalone_test.tcl -notrace -tclargs standalone_test, exiting...
make: *** [Makefile:5: sim] Error 139
(venv) (vivado2022_2) [clm61942@pc0137 standalone]$ cat hs_err_pid3310794.log 
#
# An unexpected error has occurred (11)
#
Stack:
/lib64/libc.so.6(+0x4eb80) [0x7fb99804db80]
/lib64/libc.so.6(+0xd052f) [0x7fb9980cf52f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcmalloc.so.4(__libc_calloc+0xc6) [0x7fb999d41ad6]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdiconfig.so(+0xb54f) [0x7fb993f9954f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdiconfig.so(libconfig_yyparse+0xaa2) [0x7fb993f97ce2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdiconfig.so(+0xb2d0) [0x7fb993f992d0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdiconfig.so(libconfig::Config::readString(char const*)+0xd) [0x7fb993f9a51d]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgrFile::parse()+0x108) [0x7fb999153338]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgrImpl::getMsgFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x93) [0x7fb99915eee3]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgrInstanceImpl::checkId(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x149) [0x7fb999175299]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgrInstanceImpl::ComMsgMgrInstanceImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ComMsgMgr::MsgMask, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x118) [0x7fb9991753f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgrInstance::ComMsgMgrInstance(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ComMsgMgr::MsgMask, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4c) [0x7fb99916dc2c]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(ComMsgMgr::sendMsgString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ComMsgMgr::MsgMask, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x19) [0x7fb99913f379]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_wavedata.so(SimBridge::FrontEndControl::runRestart() const+0x8b) [0x7fb95cba4d7b]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObjectImp::issueBlockingCommand(RSDData::CommandContainer const&, HDGUIStatus*, Tcl_Interp*)+0x883) [0x7fb964c15393]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simdata.so(RSDData::RSDLiveSimulationObject::issueRunRestart()+0x36) [0x7fb964c15626]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_simtcltasks.so(+0x21c4c8) [0x7fb94d67c4c8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fb9994f330f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fb993a7656f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x34bf8) [0x7fb993a77bf8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fb993a78163]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7fb993adcd2a]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2752c0) [0x7fb97f9a62c0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fb9994f330f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fb993a7656f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fb993a766a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fb993a784c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bb7f8) [0x7fb97f9ec7f8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commontasks.so(+0x2bcdff) [0x7fb97f9eddff]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xb9430f) [0x7fb9994f330f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(+0x3356f) [0x7fb993a7656f]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fb993a766a2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fb993a784c2]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_commonmain.so(+0xc3d8) [0x7fb99a1993d8]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7fb993ae32f0]
/dls_sw/FPGA/Xilinx/Vivado/2022.2/lib/lnx64.o/librdi_common.so(+0xbc17ab) [0x7fb9995207ab]
/lib64/libpthread.so.0(+0x81ca) [0x7fb997de71ca]
/lib64/libc.so.6(clone+0x43) [0x7fb998038e73]
glennchid commented 1 year ago

This turns out to be a bug in the restart function of the simulator. See, for example: https://support.xilinx.com/s/question/0D54U00006HBxxNSAT/vivado-20222-crashes-when-i-restart-the-simulation?language=en_US and other related forum posts. Resolved by removing restart function from scripts and setting runtime to all on the first pass.