homog_mem.py debug flag and max threading uxm build

[NichKu]

Hi, running uxm build, I noticed that it will produce an error unless the following is added to homog_mem.py:

_parser.add_argument('--debug', '-d', action='storetrue')

perhaps you have missed that during the last commit.

I also noticed that it will fail to build the atlas with a high number of threads. Maybe you want to add a warning that it might crash if too many cores are used. -@ 50 worked while -@ 100 failed all the time (at least for me)

[nloyfer]

Thank you for the helpful suggestions. I fixed the debug flag problem (moved it from deconv.py to homog_mem.py). About the threads issue - I couldn't reproduce it. It works for me with 100 threads. Did you get any error message/s?

[NichKu]

Regarding the treading issue, I got the following message ([...] is a private path) while generating the memoization files:

_[ wt homog ] [ GSM5652203_Heart-Fibroblasts-Z0000041X ] WARNING: all zeros! WARNING: possibly failed in [...]/pats/pats_hg19/GSM5652203Heart-Fibroblasts-Z0000041X.pat.gz - all 978 values are zero. memoization is not updated, to be safe

I got more of the same errors for other GSM_* files.

At the end I got:

_multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "[...]/.conda/pkgs/UMX/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "[...]/.conda/pkgs/UMX/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "[...]/github_repos/UXM_deconv/src/homog_mem.py", line 227, in pat2homog tmp_homog_path = wrap_cpp_tool(pat, remain_mrk, tmp_dir_l, rlen, verb, debug) File "[...]/github_repos/UXM_deconv/src/homog_mem.py", line 167, in wrap_cpp_tool subprocess.check_call(cmd, shell=True, stderr=so, stdout=so) File "[...]/.conda/pkgs/UMX/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'wgbstools homog -f --rlen 4 -b [...]/UXM_deconv/tmp_dir/l4/GSM5652342_Bladder-Epithelial-Z000000QM.8deaqbh3.bed [...]/pats/pats_hg19/GSM5652342_Bladder-Epithelial-Z000000QM.pat.gz --prefix [...]/UXM_deconv/tmp_dir/l4/GSM5652342Bladder-Epithelial-Z000000QM.8deaqbh3' died with <Signals.SIGSEGV: 11>. """

dmesg returns:

_[336324.876686] show_signalmsg: 29 callbacks suppressed [336324.876693] python3[108652]: segfault at 7fb51195d9d0 ip 00007fb535801f81 sp 00007ffd36f18990 error 4 in libpthread-2.17.so[7fb5357f9000+17000] [336325.066166] python3[108646]: segfault at 7f452c8af9d0 ip 00007f4598753f81 sp 00007fffee689670 error 4 in libpthread-2.17.so[7f459874b000+17000] [336328.202528] python3[112005]: segfault at 7fa0208a39d0 ip 00007fa03df5af81 sp 00007ffd55ae62e0 error 4 in libpthread-2.17.so[7fa03df52000+17000] [336328.840650] python3[112353]: segfault at 7ff534b0f9d0 ip 00007ff5c29b7f81 sp 00007ffd753483a0 error 4 in libpthread-2.17.so[7ff5c29af000+17000] [336329.939377] python3[112656]: segfault at 7f0fd9faa9d0 ip 00007f102464bf81 sp 00007ffcf6085110 error 4 in libpthread-2.17.so[7f1024643000+17000] [336330.271708] python3[112700]: segfault at 7f74dba649d0 ip 00007f751e105f81 sp 00007ffe5cf92680 error 4 in libpthread-2.17.so[7f751e0fd000+17000] [336330.624133] python3[112761]: segfault at 7f589746d9d0 ip 00007f58d9b0ef81 sp 00007ffea2845b30 error 4 in libpthread-2.17.so[7f58d9b06000+17000] [336330.881537] python3[112823]: segfault at 7ff3eaec79d0 ip 00007ff415568f81 sp 00007ffe95303a60 error 4 in libpthread-2.17.so[7ff415560000+17000] [336331.241338] python3[112883]: segfault at 7f3dde4329d0 ip 00007f3e40ad3f81 sp 00007fff580af5e0 error 4 in libpthread-2.17.so[7f3e40acb000+17000] [336331.521982] python3[112941]: segfault at 7fd61d43d9d0 ip 00007fd647adef81 sp 00007ffe1d81b540 error 4 in libpthread-2.17.so[7fd647ad6000+17000] [336331.633733] python3[113002]: segfault at 7fab158609d0 ip 00007fab6ff01f81 sp 00007ffca11ab750 error 4 in libpthread-2.17.so[7fab6fef9000+17000] [336331.748252] python3[113021]: segfault at 7fccf7dde9d0 ip 00007fcd68c7cf81 sp 00007ffcce685830 error 4 in libpthread-2.17.so[7fcd68c74000+17000] [336331.873550] python3[113040]: segfault at 7fcb38f339d0 ip 00007fcb91dd1f81 sp 00007ffc556accc0 error 4 in libpthread-2.17.so[7fcb91dc9000+17000] [336332.010403] python3[113056]: segfault at 7f2a567a69d0 ip 00007f2ab7644f81 sp 00007ffe062450e0 error 4 in libpthread-2.17.so[7f2ab763c000+17000]

Here I used 50 CPUs and memory should not be an issue with 500Gb. However, it seems like there might be some issues specific to me and not necessarily a bug in your code.