(not so) small tasks around swarm 3.0

frederic-mahe commented 5 years ago

Changelog for end-users

swarm 3.0 expects dereplicated fasta files and exits with an error message if this is not the case,
the only exception to that rule is when -d 0 (i.e. the number of differences is set to zero),
swarm 3.0 yields results identical to that of swarm 2.0, up to 10x faster and with half the memory footprint.
clusters are written to output files in the order they are generated, with the exception of the seeds (-w) which are sorted by decreasing abundance, and then by alphabetical order of sequences.
ported to Windows x86-64, ARM 64, and POWER8.

Under the hood:

complete rewriting of the d = 1 algorithm,
code modernization and code quality improvements for all functions (more than 1,600 fixes),
tested with state-of-the-art static and dynamic C++ analyzers,
tested on a wide range of compilers and three of the most frequent CPU architectures,
669 unit tests created, covering more than 95% of swarm's code (on a given architecture),
automatic testing of code modifications with Travis-CI,
first steps toward replicable compilations

To do

review the wiki (provide an updated version of my pipeline?),
add a link to the seminal paper by Qu et al., 2009 (algorithm similar to swarm's, unknown to us when we created swarm).

Done

[x] tag the release as v3.0.0,
[x] test the effect of compilation parameters on speed (-march=native vs. -march=x86-64),
[x] review README (mandatory dereplication, update memory consumption values, shorter README),
[x] review man page (mandatory dereplication),
[x] use code coverage results to add new tests,
[x] run cppcheck and sanitizer (see softwipe),
[x] review help (-h) output,
[x] bump up version numbers and copyright years (readme, makefile, binary, individual source files),
[x] review version (-v) output,
[x] compilation tests (gcc 4.9 to gcc-10-alpha, clang 3.8 to 9) (all good, no warning)
[x] run valgrind (automatic unit tests for each swarm usage)
[x] run fuzzing tests (three weeks of stress-tests on 12 cores, no crash detected)
[x] run fuzzing tests with address sanitizer (two weeks of stress-tests on 12 cores, no crash detected)
[x] convert companion python scripts to python 3 (support for python 2 ends this year)
review unit tests:
- [x] add missing tests,
- [x] eliminate unnecessary tmp files,
- [x] eliminate unnecessary variables,
- [x] use shorter input,
- [x] one input per test (avoid using a general input test),
- [x] eliminate obsolete bash syntax,

frederic-mahe commented 5 years ago

Help entries that could be modified (some suggestions):

 -d, --differences INTEGER           resolution (1)

 -b, --boundary INTEGER              min mass of large OTUs (3)
 -c, --ceiling INTEGER               max memory in MB for the Bloom filter (unlimited) 

 -i, --internal-structure FILENAME   write internal OTU structure to file

 -o, --output-file FILENAME          output result to file (stdout)
 -r, --mothur                        output using a mothur-like format
 -u, --uclust-file FILENAME          output using a UCLUST-like format to file
 -w, --seeds FILENAME                write OTU representatives to FASTA file

frederic-mahe commented 5 years ago

swarm's uclust format output: check whether column 3 entry C "cluster size" should be the number of amplicons or the number of reads (sum of abundances)?

EDIT

usearch tallies amplicons, not reads:

printf ">s1;size=2;\nAAAA\n>s2;size=1;\nAAAA\n" > tmp.fas
usearch7 -cluster_fast tmp.fas -minseqlength 1 -id 0.5 -uc tmp.uc
cat tmp.uc
rm tmp.*

usearch reports a cluster size of 2 amplicons, not an abundance of 3. vsearch and swarm behave like usearch.

torognes commented 5 years ago

Made the suggested changes to the help text in commit c07a0d440858309b670a6dd0657e4847c5c2fd73.

torognes commented 4 years ago

Is it ok if I set up Travis CI to compile (and possibly test) Swarm, as done for vsearch?

frederic-mahe commented 4 years ago

Is it ok if I set up Travis CI to compile (and possibly test) Swarm, as done for vsearch?

Yes, that sounds like a good idea. Can Travis CI fetch tests from the swarm-tests repository?

torognes commented 4 years ago

Can Travis CI fetch tests from the swarm-tests repository?

Yes, I think so.

torognes commented 4 years ago

Travis CI will now automatically compile and test swarm3 after any commit is pushed or any pull request is submitted.

The status can be seen here: https://travis-ci.org/torognes/swarm

It fails if it compiles with an error or if any of the tests fail.

It is compiled using g++ version 7.4.0 on Ubuntu 18.04.3 (bionic) linux.

The swarm-tests repo is automatically cloned from its source.

It all seems to work fine now after a series of modifications.

There is a badge on the front page (in README.md) showing the latest status.

This only applies to the swarm3 branch.

torognes / swarm