Basic performance metrics

[d-kk]

Hi,

We would like to use this dataflow and it would be great if we could know what type of machine you used (especially nproc and memory) and how long it took to run the dataflow. We'll spec our VM accordingly.

Thanks,

David

[taylorreiter]

Hi David, sorry for taking so long to respond, I missed the notification for this issue.

• I used a machine with 64 threads for the initial clustering. I believe it took ~3 days
• Some of the subsequent steps spiked to ~500 gb of ram, so I used an r5a.16xlarge for these (I believe the csvtk join and fold steps).

I was...very sloppy with my benchmarking of this workflow. I tried a million an twelve things before I settled on this workflow, in part because of bugs in taxonkit that have now been resolved.

[d-kk]

Thanks appreciate the info. 🙂