Rust marco dev

[mmolari]

I fixed some bugs in the implementation of minimizer sketching and added nontrivial tests. The bugs were simply due to the fact that Julia has 1-based indexing, while rust is 0-based. I hope I catched caught all of them. The test were generated by running the original julia code on a random 100-nucleotide sequence, and getting the resulting minimizers.

[ivan-aksamentov]

Oh cool!

My only worry is that positions in the result look suspiciously like u64 max :) Did we hit some edge case here?

I'd merge, but I see that you've added bunch of directories from the Julia branch to gitignore. They are not relevant to Rust version and this might bite us later - I will definitely miss that some files did not get committed at some point in the future, and might loose some of the work progress.

Here is a potential alternative: I clone rust branch as a separate directory, this way I can have both Julia and Rust versions available at the same time, and I don't need to switch and worry about untracked directories specific to one or the other version. It's a complete rewrite, so switching the same local repo is very inconvenient. Do you think you can also adopt this strategy and revert the gitignore entries? Otherwise you could also try to add them to your local gitignore file called .git/info/exclude, where .git is the directory in the root of the checked out project.

[mmolari]

Thanks for all of the feedback!

• concerning the .git file you are right. I removed the changes and I cloned the repository again in another folder to keep things clean.
• concerning the position in the results, it's because in the julia version the position attribute is a bitpacked container for sequence id (first 32 bytes) locus (next 31 bytes) and strandedness (first bit). We might want to simplify this later on but for now I left it, so that we can compare directly with the Julia output.

I also added some more corrections. They are all due to the fact that:

• julia has 1-based indexing while rust has 0-based indexing
• iterators a:b in julia include the extremes while in rust a..b excludes b. These corrections should only be relevant in edge cases. I could not trigger an error with small examples.

[mmolari]

I also fixed some small bugs in the mash_distance function and added some non-trivial tests. I think we might have still some small conceptual problems here:

• this is not exactly equivalent to the julia implementation because of order dependence. Because of the way the distance is calculated, the order of sequences matters, see this line. In julia the order is given by the dictionary in which sequences and names are stored while in rust it is given by the order in which graphs are passed to the function. It's not just a matter of reordering the numbers, it might give different numbers because the denominator in the distance division might be different, possibly by much for sequences of different length. I prefer rust way, it's more explicit than the order of keys in a dictionary. Eventually we might want to add a small change in this function to eliminate the order dependence, but for now it reproduces julia's results modulo this order dependence.
• the second problem is that tests are strongly reliant on the hash function and the minimizer sketching. If we change any of these functions we will break all of the tests, e.g. if we decide to use a different hash function. I don't know if this is necessarily a problem, maybe just something to keep in mind.