search

nanoporetech / dorado

Oxford Nanopore's Basecaller
https://nanoporetech.com/
Other
493 stars 59 forks source link

V8 seems to be much slower #1038

Open weishwu opened 1 week ago

weishwu commented 1 week ago

I see other users reported the same issue: https://github.com/nanoporetech/dorado/issues/1029

The reply said it was because of the sup@v5 modification model. However, I've been using this model for a while, but the basecalling speed seems to get much slower after I switched to Dorado v8. With the older version typically one basecalling job took about 5 days. With v8, my jobs are still running after 6 days. I'm using the default parameter setting. Is there anything I can tweak to make it faster?

HalfPhoton commented 1 week ago

Hi @weishwu, I don't believe that the issue you've cited is the same.

Can you compare the reported batchsize and samples/s from v7 and v8 and tell us what hardware you're using?

Best regards Rich

weishwu commented 1 week ago

Now I think it may be because there are significantly more reads in the new data.

Another question: I read from this thread that different GPU architecture may affect reproducibility. I have these GPU nodes in the cluster I use:

My basecalling jobs are spread across these nodes. Should I worry about reproducibility? Thanks.

HalfPhoton commented 6 days ago

Hi @weishwu,

Basecalling can yield slightly different results on different GPUs due to minor discrepancies in the arithmetic used. These differences will typically appear in areas of low quality (low Q-score) and should not affect high confidence calls. As such, for the best reproducibility we recommend using the same hardware if possible. However, if it's a matter of needing to reproduce the same calls at a later date - the GPU used is recorded in the output BAM file which will aid you in keeping track of which GPU was used across different jobs on your cluster for example.

Best regards, Rich

weishwu commented 3 days ago

Hi @HalfPhoton I remove the reads that have min baseq lower than 10, but this won't do anything with the low baseq portions in the remaining reads. Our analysis goal is to measure allele-specific methylation and then examine the variability between individuals. I wonder if I need to redo the basecalling for my samples by using the same GPU type. All the GPU nodes I ran jobs on are NVIDIA. It's just that some are A100 and some are V100. I use this model by the way: dna_r10.4.1_e8.2_400bps_sup@v5.0.0 with 5mCG_5hmCG.