Command terminated by signal 4

[colindaven]

Hope this helps, thanks for the nice tools.

Setup - current Singularity image based on docker.

Versions

sing exec image.sif seqwish -v
v0.7.6

sing exec image.sif pggb --version
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
fatal: not a git repository (or any of the parent directories): .git
(pggb requires git for versioning, but git likely not in the docker container from which this singularity container was built).

sing exec image.sif smoothxg -v
v0.6.5

Failed run:

log pangenome.fasta.gz.339b688.e34d4cd.dcef834.smooth.08-05-2022_09:48:03.log

Log from one machine - new, two years old (what cpu specs do you need?)
uname -a 
Linux XXXX 4.15.0-184-generic #194-Ubuntu SMP Thu Jun 2 18:54:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

 ** This machine only has avx, not avx2 like the lower machine has. **

processor       : 39
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
stepping        : 4
microcode       : 0x42e
cpu MHz         : 1200.188
cache size      : 25600 KB
physical id     : 1
siblings        : 20
core id         : 12
cpu cores       : 10
apicid          : 57
initial apicid  : 57
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips        : 5602.08
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Successful run:

log pangenome.fasta.gz.339b688.e34d4cd.dcef834.smooth.08-05-2022_09:42:09.log

Different (older?) machine BUT with newer kernel (note different kernel
Linux YYYYYY  5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

processor   : 39
vendor_id   : GenuineIntel
cpu family  : 6
model       : 79
model name  : Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
stepping    : 1
microcode   : 0xb00003e
cpu MHz     : 1201.349
cache size  : 25600 KB
physical id : 1
siblings    : 20
core id     : 12
cpu cores   : 10
apicid      : 57
initial apicid  : 57
fpu     : yes
fpu_exception   : yes
cpuid level : 20
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips    : 4801.28
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

[colindaven]

Could it be the kernel looking at these further tests?

works on another machine with avx2 and 5.4.0-117 kernel
Linux ZZZ 5.4.0-117-generic #132-Ubuntu SMP Thu Jun 2 00:39:06 UTC 2022 x86_64 GNU/Linux

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca

fails on another machine without avx2 and with 4.15 kernel
Linux AAA 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 GNU/Linux

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d

works on last machine **without avx2 but with 5.4 kernel**
uname -a
Linux BBB167 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 GNU/Linux

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm pti arat

[AndreaGuarracino]

Hi @colindaven, thank you for all the detailed information. I have hypothesis. Would you please add --run-abpoa in the failing pggb run and check if it still fails?

[colindaven]

Yes, this now succeeds - at least with my tiny test dataset which reproducibly failed before. Will test further with bigger data.

[AndreaGuarracino]

Thanks! Just to be sure, the command line

pggb -i splits/pangenome.fasta.gz -V INO1_org:. -t 16 -T 14 -o pangenome -Y . -S -m -M -P 1,19,39,3,81,1 -p 95 -s 200 -k 47 -n 6

(or similar) fails in some systems. Instead, if you add --run-abpoa to it, the command line works always.

Is this the case?

[colindaven]

Let's put it this way - I've added --run-abpoa to the snakemake pipeline I am using.

I have 3 test datasets. All 3 were failing beforehand on most machines (a colleague also reported a signal 7 failure bug on a further dataset, which I could not reproduce).

Since introducing the --run-abpoa change I cannot see failures on any datasets or machines (running either machinewise using local, or distributing across a cluster).

I will let you know if I see any further errors, but at the moment adding seems --run-abpoa has corrected the issue for me.

Thanks!

[AndreaGuarracino]

Glad to read it! @colindaven, I would have one last request, if I may: could you please try the latest Docker/ image (from today, 08/17/2022) without the --run-abpoa flag? I've made a fix that hopefully solves the problem.

[AndreaGuarracino]

Feel free to open this again if the problem pop-ups again.

[colindaven]

Still testing as you suggested, but haven't seen the error again yet. Thanks.