search

onecodex / finch-rs

A genomic minhashing implementation in Rust
https://www.onecodex.com
MIT License
92 stars 8 forks source link

multi-threaded mash_files #37

Closed wwood closed 4 years ago

wwood commented 4 years ago

Hi,

Thanks for writing finch, I'm finding it quite useful.

Personally I've not used crossbeam before, but as I understand it should be relatively straightforward to multi-thread the mash_files function, where each thread is given a file path (ie the for loop). Is anything like that on the cards, or would a sufficiently well written pull request be likely to pass muster given it would introduce further cargo dependencies?

Thanks, ben

bovee commented 4 years ago

I agree it should be fairly straightforward to multi-thread; it hasn't been in the plans so far, but it's a really good idea and I don't think we're at all adverse to including crossbeam as a dependency. We're happy to take a PR (or I'll probably take a stab at it at some point as we move towards v0.3).

And thanks for the nice comments!

bovee commented 4 years ago

I just put a commit on my current "version 0.3" working branch to do this. Hopefully this can get merged within the next month or so.

This should speed up any downstream Rust code using mash_files, but unfortunately it's not going to provide a speed-up for the CLI yet (since the CLI calls mash_files with one file at a time!) I'm going to try to clean up the CLI code too for version 0.3 and hopefully it's fairly easy to make it use mash_files with all of the relevant file names.

wwood commented 4 years ago

Excellent, thanks for that. Personally the cli doesn't matter since I'm using it as a library, but good to hear it's in the pipeline.

wwood commented 4 years ago

Hi,

Any news on a new release? I'm hoping to release some code and would be good to depend on an official version. Thanks.

bovee commented 4 years ago

Hi, sorry I missed this! These changes should have been pushed a few months ago when we released v0.3.0; are you not seeing that version?

wwood commented 4 years ago

Ah, my mistake, I didn't realise. All good.

bovee commented 4 years ago

No problems! I just realized our Homebrew formula was out-of-date so this was a useful reminder.