dataprep high coverage sample

[rania-o]

Hello,

I used the dataprep command for my in vivo sample with a depth around (1400X, ~9000 reads) and it worked fine +1

xpore dataprep --eventalign /nanopolish/output.eventalign.txt --out_dir results --readcount_min 20 --readcount_max 1000000

But for my control sample with more depth (>500,000 reads) I got this error after a long time running :

xpore dataprep --eventalign /nanopolish/output.eventalign.txt --out_dir ctrl_results --readcount_min 20 --readcount_max 1000000

Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/queues.py", line 240, in _feed send_bytes(obj) File "/usr/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647

So I tried it again with --readcount_max 50000 and I got the same error. Can Xpore handle samples with high coverage?

Thank you Rania

[yuukiiwa]

Hi Rania (tagging you here @rania-o),

Our lab works with samples with >1M reads and has not encountered any problem running xpore. I suspect that you might have exceeded your machine's maximum memory when running xpore.

Thanks!

Best wishes, Yuk Kei

[rania-o]

Hi @yuukiiwa

Thank you for your fast reply. I've just checked, and we have 250 G of memory (available), do you think it's not enough to run Xpore for ~600 000 reads ?

Rania

[yuukiiwa]

Hi Rania (@rania-o),

Do you have 250Gb storage or memory? Here is the command to see how much memory available on your computer:

free -g -h -t

Thanks!

Best wishes, Yuk Kei

[rania-o]

Hi @yuukiiwa

Yes, it's the command I used, here is the output : total used free shared buff/cache available Mem: 251G 8.6G 25G 55M 217G 241G Swap: 8.0G 469M 7.5G Total: 259G 9.0G 33G

Do you think it's enough ? Thanks, Rania

[yuukiiwa]

Hi Rania (@rania-o),

Which python version do you use? I googled, and here it says that python 3.8 gives this kind of error

Thanks!

Best wishes, Yuk Kei

[rania-o]

Hello  @yuukiiwa

Thanks for your reply. I use : Python 3.6.9, but I saw that is python 3.6 which gives the error and in python 3.8 they delete it.

Rania