Open hasindu2008 opened 1 month ago
Hi @hasindu2008 ,
It looks reasonable, do you know the number of megabytes per second of signal you receive from the loading? This is a more interesting performance metric as any operation you run involving the sample data will bottleneck on decompression - and potentially the disk IO speed depending on the speed of your processor.
If you aren't hitting your disk's max read speed I would suggest it could be more optimal.
Depending on the size of the pod5 files and the size of the selection you are making, you could explore passing the reads from an entire batch as one async operation to reduce overhead of the threadpool - but you would need to configure this based on profiling of your use case.
Hope that helps,
Hi
Could you clarify what you meant by "megabytes per second of signal" - before decompression or after compression? effective speed for the subsets or what is observed through something like iostat?
The CPU is maxing out, so I guess the overhead of the threadpool is negllegable and optimising this would not make a difference I guess?
@0x55555555 to add further:
As per iostats, it is reaching like 300 MB/s:
r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
3538.60 0.00 336.05 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.16 97.25 0.00 0.04 15.36
r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
3122.80 0.00 297.02 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.15 97.40 0.00 0.04 14.00
r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
3297.00 0.00 313.64 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.13 97.41 0.00 0.04 12.48
r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
3346.00 0.00 317.91 0.00 0.00 0.00 0.00 0.00 0.16 0.00 0.13 97.29 0.00 0.04 12.72
r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
3444.20 0.00 328.32 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.14 97.61 0.00 0.04 13.12
For a typical SSD, sequential read speed is around 500MB/s, so I guess for random access 300MB/s is likely to be closer to the max? I am not sure how random access in POD5 is implemented under the API and thus what kind of size each random access is, but you could answer this better I presume.
The size of my POD5 file is 741G which contains a whole PromethION sample. The size of the selection I make at a time is around 1000 reads, imagine getting reads belonging to a certain gene region. If you have any code examples of the suggestion you mentioned about "passing the reads from an entire batch as one async operation to reduce overhead of the threadpool", we could try that too.
Thanks.
If your CPU is maxing out, its more likely you are bottlenecking on decompression of signal - which you could confirm using perf
tools, but you're right, for a single SSD system, this might be the max you can achieve.
It looks like you are achieving a reasonable level of performance though - it'll depend on the downstream application of the signal if you need to optimise further.
Thanks,
I do not quite understand how the random access in pod5 works. Is there an index or something? If not, why not?
Hi pod5 developers,
I am trying to do some random access to a POD5 file, a small batch of around 1000 at a time. For instance, say we want to grab the reads for a small gene region at a time. Looking at the Dorado code, we have written the following piece of code that would be exploiting multiple threads.
We get like ~316 reads per second on a system with SSD. We are running with 40 threads and the CPU utilisation seems to be good.
Could you advise if this looks right interms of maximally getting the expected performance?