search

GoekeLab / xpore

Identification of differential RNA modifications from nanopore direct RNA sequencing
https://xpore.readthedocs.io/
MIT License
132 stars 22 forks source link

weird struct.error when running xpore-diffmod #28

Closed pabloacera closed 9 months ago

pabloacera commented 3 years ago

I am running xpore-diffmod with a reasonably large files and I get this error in the middle of the execution. Have you seen this error before? 

[pm1122@gadi-hmem-clx-0013 WT2_RNAAB056712]$ /home/249/pm1122/raijin_home_2019-11-15/pm1122/lib/bin/xpore-diffmod --n_processes 5 --config /g/data/xc17/pm1122/yeast/yeast_config_test.yml --save_models
Using the signal of unmodified RNA from /home/249/pm1122/raijin_home_2019-11-15/pm1122/lib/lib/python3.6/site-packages/xpore/diffmod/model_kmer.csv
18 ids to be testing ...
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr1.hdf5 chr1
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr10.hdf5 chr10
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr11.hdf5 chr11
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr12.hdf5 chr12
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr13.hdf5 chr13
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr14.hdf5 chr14
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr15.hdf5 chr15
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr16.hdf5 chr16
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr2.hdf5 chr2
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr3.hdf5 chr3
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr4.hdf5 chr4
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr5.hdf5 chr5
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr6.hdf5 chr6
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/queues.py", line 240, in _feed
    send_bytes(obj)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 204, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 397, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/scplasm1.hdf5 scplasm1
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr8.hdf5 chr8
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr9.hdf5 chr9

I previously ran xpore-diffmod with even larger files and the error happened before processing chromosome 1. To be honest there is not much info about this error online and I have little idea about it, but I am guessing that it has to do with some kind of memory issue or loading of big files? maybe the data.json??

Just in case you can shed some light here! thanks!

pabloacera commented 3 years ago

This is an issue related to the upper limit size of objects share between processes, this issue is fixed in python 3.8 or above. Cheers!

ploy-np commented 3 years ago

@pabloacera Thank you very much. Yes, it was the memory issue. I've encountered this before when I used python 3.7 :(. I've changed the python version required for this package to >=3.8 already. Thanks again!