search

refresh-bio / clusty

GNU General Public License v3.0
4 stars 1 forks source link

Clusty is slow to read long inputs #2

Open apcamargo opened 4 days ago

apcamargo commented 4 days ago

I generated a large similarity table from ~1 million genomes using sourmash'es branchwater and tried to cluster it with Clusty. Clusty didn't finish reading the input after 6 hours, while pyLeiden finished reading everything within ~30 min.

Is there a reason Clusty is taking so much time to read the input?

agudys commented 4 days ago

Hello!

We manged to run Clusty on distance tables having tens of gigabytes so it seems there is some issue which made Clusty hung on your dataset. How large is your data file and how many distances it contains? Could you please provide me with at least part of it?

Best, Adam

apcamargo commented 3 days ago

I don't have the original input anymore, but a filtered version (which Clusty is also taking a long time to read) is ~34GB with 401,294,724 lines (not counting the header).

Here's a sample with 500k lines.