Closed dkoslicki closed 2 years ago
Disconcertingly, when adding a print(k)
here, it appears to bounce back and forth between the original k
passed in from the CLI and 3*k
.
@bluegenes Is it possible that there's some oddity about when and where the -k
is (or isn't) converted to 3*k
and/or k/3
?
yes, the Rust layer uses k*3 internally for protein, while the Python layer uses k. Looks like you found a place we didn't do a translation :)
note, see https://github.com/sourmash-bio/sourmash/issues/1383
will look into it!
whew, that was a journey.
Turns out that we are not properly handling pickle-based serialization of non-DNA MinHash objects. This is a general problem; here it was triggered by the use of serialization to communicate objects between different processes.
In slightly more detail,
Adding to this, it looks like numpy is implicitly/silently using the __reduce__
protocol.
Anyhoo. Fixed in https://github.com/sourmash-bio/sourmash/pull/2265.
whoops, looks like it's multiprocessing that was correctly using the pickle protocol, which is even documented properly; and it was just stumbling over broken sourmash code. So I made this more complicated than it needed to be in the previous comment 😆
I have a protein signature:
When running:
I get the following error:
This does not occur in the serial version:
Nor in the parallelized, non-ANI version:
It appears that it doesn't like the k-mer size
self_mh.ksize
that is being passed as an argument here. I hypothesize this is due to a k-mer size not being multiplied by 3 somewhere. Eg. similarity does not appear to callself_mh.ksize
anywhere, yet jaccard_ani does.Commenting out this line of code makes the error go away. But I don't know the ramification of removing this assertion to other parts of the source code.