search

aiiie / cram

Functional tests for command line applications
GNU General Public License v2.0
199 stars 51 forks source link

Apparently spurious test failures #5

Closed 0xaf1f closed 8 years ago

0xaf1f commented 8 years ago

One of our Debian packages uses cram tests (v0.6) which fail on some architectures and pass on others. Where the tests fail, I see no difference between the output and input. See, for example, the build log for the package on 32-bit x86 at https://buildd.debian.org/status/fetch.php?pkg=python-pbh5tools&arch=i386&ver=0.8.0%2Bdfsg-2&stamp=1447588876 where failures look like:

 ## Selection
   $ cmph5tools.py select --groupBy Barcode \
   > --where "(Barcode == 'F_42--R_42') | (Barcode == 'F_10--R_10')" $INCMP
   $ cmph5tools.py merge --outFile merged.cmp.h5 F_42--R_42.cmp.h5 F_10--R_10.cmp.h5
   $ cmph5tools.py stats --what "Count(Reference)" --groupBy Barcode merged.cmp.h5
-                         Group                    Count(Reference)
-                    F_10--R_10                                  62
-                    F_42--R_42                                  76
+                      Group                    Count(Reference)
+                 F_10--R_10                                  62
+                 F_42--R_42                                  76

The tests pass just fine on 64-bit x86. I'm not sure if there's some whitespace problem or if there's a way to ignore that. For now, I have to ignore the cram test results during the build for this package because of these spurious failures, but I think this should be sorted out at some point.

Many thanks and regards

0xaf1f commented 8 years ago

I hadn't thought to use sed or some regular expressions. Sorry for the noise..