bug in abundance value parsing?

[frederic-mahe]

so, this is weird. Swarm accepts arbitrary large abundance values, but It seems to fail to parse abundance values that are exact powers of two, for powers greater than 31 (so, 2^32, 2^33, 2^34, ...). For example:

printf ">s1_%d\nA\n" $(( 1 << 31 )) | swarm
success

printf ">s1_%d\nA\n" $(( 1 << 32 )) | swarm
fail

printf ">s1_%d\nA\n" $(( (1 << 32) + 1 )) | swarm
success

Maybe there is a problem with the header parsing function and the way atol() works?

[lczech]

This is an overflow issue. abundance is defined as unsigned int, which it is usually 32 bit.

Thus, any power of two greater than that overflows to 0. Any other number greater than that will overflow to some non-zero number, which makes the program run through, but with wrong abundances. If you print the abundance for ( 1 << 32 ) + 1, you get 1. This is a valid abundance, but not want you wanted.

(For that reason, it is probably a good idea to use size_t for all counting.)

[frederic-mahe]

Hum, I am not sure it is an overflow. Swarm does return the correct abundance value, even if it is larger than 2^32:

echo $(( (1 << 34) - 1 ))
printf ">s1_%d\nA\n" $(( (1 << 34) - 1 )) | swarm 2> /dev/null                                                                                                                         
17179869183
s1_17179869183

[lczech]

I have not looked into the part, that does the output, but it seems it simply prints the string that was used as input for this, without conversion to int.

The conversion from string to int when reading the database in db.cc however stores the result in an unsigned int, so the overflow is happening at this point. The variable is defined in seqinfo_s struct in swarm.h, where abundance is stored as unsigned int, so it cannot hold values of 1 << 32 or more, given that it is a 32 bit number (on most platforms).

[torognes]

The abundance values should probably instead be represented by an unsigned 64-bit long so that they can hold values up to 2^64-1 = 1.8e19 instead of just 2^32-1 = 4.2e9. I guess abundances above 4.2e9 are possible.

[lczech]

And I insist on the opposite ;-)

In db.cc, line 374

seqindex_p->abundance = atol(seqindex_p->header + pmatch[2].rm_so);

add a print statement like this

std::cout << "\n\nseqindex_p->abundance = " << seqindex_p->abundance << "\n\n";

and add #include <iostream> in the beginning of the file. Then try your test cases again.

As said, exact powers of two overflow to 0, all others overflow to some value in the range [ 1, 2^32 - 1 ], which are then used as valid abundance values, so the program seems to work, but actually uses incorrect values.

I guess you are still seeing the correct abundance in the output file because the literal string is used there, instead of a back-conversion from int to string. Maybe @torognes can confirm this.

[frederic-mahe]

You are right. I deleted my comment a few seconds after posting it, when I'll realized that swarm was only returning a "string" version of the abundance value.

[frederic-mahe]

Thanks for your comments. Data sets are not that big yet, so it is not urgent to make that change. But we will get there eventually, so I am going to mark that issue as a feature for swarm 3.0.

[torognes]

Fixed. Now allows 64-bit values.