BGSpiegl
commented
1 week ago Hi Maria,
Thank you for helping me improve GCparagon and learn more about the computing environments used by others. Good news: it seems only plotly can't write a plot file and nothing is wrong with the BAM file output per se.
This seems to be an issue with plotly's Kaleido backend. Kaleido not having access to the /tmp directory on your computing system causes the creation of the Kaleido process to fail:
[0905/145741.927970:ERROR:platform_shared_memory_region_posix.cc(250)] Creating shared memory in /localscratch/astanciu/1830995/.org.chromium.Chromium.cM7e3N failed: No such file or directory (2)
[0905/145741.928000:ERROR:platform_shared_memory_region_posix.cc(253)] Unable to access(W_OK|X_OK) /localscratch/astanciu/1830995: No such file or directory (2)
This can be the case on a very restrictive cluster environment which seems to be the case for you. (see kaleido github issue: https://github.com/plotly/Kaleido/issues/90)
I will try to add the solution presented in the issue: https://github.com/plotly/Kaleido/issues/90#issuecomment-839778120 I will come back to you once the new image is available for download.
BR Benjamin
mariaalexandrastanciu
Hi,
If I try to run GCParagon with --output-bam flag, I get the following error:
WARNING - could not save figure 'NIPT-PRL-009-25-BL-Px_S16.markDup.GCcontent_RefVsReconstructed' in 'png' image format! Continuing .. | 2024-09-05 14:57:40,808 - GCparagon - INFO - Actual cumulative reconstruction error of the fragment GC content distribution of the reference genome: 4.73% | 2024-09-05 14:57:40,809 - GCparagon - INFO - Saving statistic matrix 'NIPT-PRL-009-25-BL-Px_S16.markDup_simulated_attributes_matrix.txt.gz' to output directory | 2024-09-05 14:57:41,245 - GCparagon - INFO - Saving statistic matrix 'NIPT-PRL-009-25-BL-Px_S16.markDup_observed_attributes_matrix.txt.gz' to output directory | 2024-09-05 14:57:41,577 - GCparagon - INFO - Mask computation: used a minimum threshold of 10 for total counts of individual GC-base count-fragment length combinations. Loading statistic matrix from /globalscratch/ulb/bctr/astanciu/tmp/NIPT-PRL-009-25-BL-Px_S16.markDup/NIPT-PRL-009-25-BL-Px_S16.markDup_observed_attributes_matrix.txt.gz | 2024-09-05 14:57:41,990 - GCparagon - ERROR - The following exception occurred when trying to process BAM file /globalscratch/ulb/bctr/astanciu/Pearl/bams/NIPT-PRL-009-25-BL-Px_S16.markDup.bam: Traceback (most recent call last): File "/opt/conda/envs/GCparagon/bin/gcparagon", line 8, in
sys.exit(main())
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3224, in main
(mask_path, weights_mask)) = compute_gc_bias_parallel(
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 1632, in compute_gc_bias_parallel
trimmed_dimensions = consolidate_results(
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 1171, in consolidate_results
plot_fragment_length_dists(matrix_data_frame=None, matrix_file_list=[observed_attributes_matrix_sum_path],
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/GCparagon/utilities/plot_distributions.py", line 209, in plot_fragment_length_dists
length_distribution_fig.write_image(out_file)
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/plotly/basedatatypes.py", line 3835, in write_image
return pio.write_image(self, *args, **kwargs)
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 266, in write_image
img_data = to_image(
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 143, in to_image
img_bytes = scope.transform(
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/kaleido/scopes/plotly.py", line 153, in transform
response = self._perform_transform(
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/kaleido/scopes/base.py", line 293, in _perform_transform
self._ensure_kaleido()
File "/opt/conda/envs/GCparagon/lib/python3.10/site-packages/kaleido/scopes/base.py", line 198, in _ensure_kaleido
raise ValueError(message)
ValueError: Failed to start Kaleido subprocess. Error stream:
[0905/145741.914500:WARNING:resource_bundle.cc(431)] locale_file_path.empty() for locale [0905/145741.922044:WARNING:resource_bundle.cc(431)] locale_file_path.empty() for locale [0905/145741.922633:WARNING:resource_bundle.cc(431)] locale_file_path.empty() for locale [0905/145741.922808:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 0 [0905/145741.927970:ERROR:platform_shared_memory_region_posix.cc(250)] Creating shared memory in /localscratch/astanciu/1830995/.org.chromium.Chromium.cM7e3N failed: No such file or directory (2) [0905/145741.928000:ERROR:platform_shared_memory_region_posix.cc(253)] Unable to access(W_OK|X_OK) /localscratch/astanciu/1830995: No such file or directory (2) Received signal 6
0 0x558ec4a33d79 base::debug::CollectStackTrace()
1 0x558ec49b1633 base::debug::StackTrace::StackTrace()
2 0x558ec4a3395b base::debug::(anonymous namespace)::StackDumpSignalHandler()
3 0x154517064520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
4 0x1545170b89fc pthread_kill
5 0x154517064476 raise
6 0x15451704a7f3 abort
7 0x558ec49df30a base::internal::OnNoMemoryInternal()
8 0x558ec49df329 base::(anonymous namespace)::OnNoMemory()
9 0x558ec49df319 base::TerminateBecauseOutOfMemory()
10 0x558ec49c90ab base::FieldTrialList::InstantiateFieldTrialAllocatorIfNeeded()
11 0x558ec49c9239 base::FieldTrialList::CopyFieldTrialStateToFlags()
12 0x558ec3524f82 content::GpuProcessHost::LaunchGpuProcess()
13 0x558ec3523910 content::GpuProcessHost::Init()
14 0x558ec35236c2 content::GpuProcessHost::Get()
15 0x558ec3940b6e base::internal::Invoker<>::RunOnce()
16 0x558ec49f7306 base::TaskAnnotator::RunTask()
17 0x558ec4a08cf6 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
18 0x558ec4a089ea base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
19 0x558ec4a55899 base::MessagePumpLibevent::Run()
20 0x558ec4a0959b base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
21 0x558ec49e29bd base::RunLoop::Run()
22 0x558ec33caf18 content::BrowserProcessSubThread::IOThreadRun()
23 0x558ec4a21874 base::Thread::ThreadMain()
24 0x558ec4a43caa base::(anonymous namespace)::ThreadFunc()
25 0x1545170b6ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
26 0x154517147a04 clone
r8: 0000154516a08fc0 r9: 0000032063b9f520 r10: 0000000000000008 r11: 0000000000000246 r12: 0000000000000006 r13: 0000000000000016 r14: 0000032063b04cc0 r15: 0000032063bb2a40 di: 00000000001ed1d5 si: 00000000001ed1db bp: 00000000001ed1db bx: 0000154516a0a640 dx: 0000000000000006 ax: 0000000000000000 cx: 00001545170b89fc sp: 0000154516a08ef0 ip: 00001545170b89fc efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated.
My call is: singularity run -B /globalscratch/ulb/bctr/astanciu/ gcparagon_0.6.8-ubuntu-22_04-container_latest.sif --bam $bamFile --out-dir $outDir --threads 8 --output-bam --temporary-directory /globalscratch/ulb/bctr/astanciu/tmp
Can you help me with it? I don't understand why it is looking for files in /localscratch/ since I specified another tmp directory.
Best regards, Alexandra