Errors at multiprocessing and matplotlib

[hyunyong-koh]

Hi, Dr. Yang. I am trying to test DeepMosaic with files in demo after launching, it shows error below:

$ deepmosaic-draw.py -i /hdd1/DeepMosaic/demo/input.txt -o ./ -a /hdd1/annovar/

NOTICE: Output files are written to ./input.variant_function, ./input.exonic_variant_function NOTICE: Reading gene annotation from /hdd1/annovar/humandb/hg19_refGene.txt ... Done with 72567 transcripts (including 17617 without coding sequence annotation) for 28263 unique genes NOTICE: Processing next batch with 4 unique variants in 4 input lines NOTICE: Output file with variants matching filtering criteria is written to ./input.hg19_gnomad_genome_dropped, and output file with other variants is written to ./input.hg19_gnomad_genome_filtered NOTICE: Processing next batch with 4 unique variants in 4 input lines NOTICE: Database index loaded. Total number of bins is 28127612 and the number of bins to be scanned is 4 NOTICE: Scanning filter database /hdd1/annovar/humandb/hg19_gnomad_genome.txt...Done

multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, *kwds)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/hdd1/DeepMosaic/deepmosaic/featureExtraction.py", line 120, in multiprocess_iterator fig1.savefig(image_file) File "/usr/lib/python3/dist-packages/matplotlib/figure.py", line 2180, in savefig self.canvas.print_figure(fname, **kwargs) File "/usr/lib/python3/dist-packages/matplotlib/backend_bases.py", line 2021, in print_figure canvas = self._get_output_canvas(format) File "/usr/lib/python3/dist-packages/matplotlib/backend_bases.py", line 1961, in _get_output_canvas raise ValueError( ValueError: Format 'jpg' is not supported (supported formats: eps, pdf, pgf, png, ps, raw, rgba, svg, svgz) """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/hdd1/DeepMosaic/deepmosaic/./deepmosaic-draw.py", line 4, in if name=='main': main() File "/hdd1/DeepMosaic/deepmosaic/featureExtraction.py", line 230, in main results = pool.map(multiprocess_iterator, all_variants, 8) File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get raise self._value ValueError: Format 'jpg' is not supported (supported formats: eps, pdf, pgf, png, ps, raw, rgba, svg, svgz)

The issue matplotlib seems to be resolved by re-installing pillow==6.0 (my pillow==7.0 and matplotlib==3.1.2.) Could you help me resolving multiprocessing issues? I could see some other StackOverFlow or boards for similar thing, but there is no clear resolution. Thank you!

[shishenyxx]

Hi khy900712,

Thank you for your question, although it seems to be a systematic issue, I can provide some suggestions (not sure it will work). I noticed that you are using Python 3.8. Maybe you can test it with Python 3.7. We are working to provide some docker images to make version control easier.

Best,

Xiaoxu

[hyunyong-koh]

Thanks, Xiaoxu. I will try it in Python 3.7. I am thankful if you offer some docker image. Looking forward to seeing it. Best wishes, Hyunyong (Howard)

[shishenyxx]

Thanks, Xiaoxu. I will try it in Python 3.7. I am thankful if you offer some docker image. Looking forward to seeing it. Best wishes, Hyunyong (Howard)

Hi Hyunyong,

As most of the HPCs don't allow admin privilege, we made a singularity image. The singularity image (https://cloud.sylabs.io/library/arzoopatel5/deepmosaic/deepmosaic) is published today and DeepMosaic is updated to 1.0.2. We also updated the documentation accordingly.

Best,

Xiaoxu