search

radiasoft / zgoubi

Git repo for zgoubi source code
https://sourceforge.net/projects/zgoubi/
GNU General Public License v2.0
9 stars 3 forks source link

Debug issue with failing tests. #86

Closed dtabell closed 4 years ago

dtabell commented 4 years ago

A number of tests pass in the VM, but not on Travis. Why?!?!? What's the problem? How can we fix it?

dtabell commented 4 years ago

For the record, I did some initial work on this issue in branch fix-common-blocks. There I saw two tests failing: chromPSR and FITandTWISS.

dtabell commented 4 years ago

Finally figured out that we were not seeing the expected out on the Travis-CI reports because the line

set -o errexit

caused the script to exit before reporting any results.

dtabell commented 4 years ago

Next problem fixed: the default ndiff does handle 1.25, as a number. Using the flag -separators '[ \t,()]' fixes this and related issues. The test chromPSR now passes. Just one left.

dtabell commented 4 years ago

It turns out the -separators flag also makes white space significant. This is a problem when numbers are different lengths (as are 0.00000 and 1.23456E-17), so I avoid this problem by instead using sed to first convert all commas to spaces and not use -separators.

Next problem to solve: What to do when a comparison is made between, say, 1.234e-16 and -5.123e-17? It certainly seems that both these values should be considered zero, and therefore equal. If we set an appropriate absolute-error tolerance, ndiff will ignore such differences. For most comparisons, however, we want to set a relative-error tolerance.

ndiff has flags for -relative-error and -absolute-error, but it ORs the comparisons, rather than ANDs them. For now I've dealt with this issue by using sed to drive tiny numbers (magnitude 1e-14) to zero before making the comparisons, but I'm not entirely happy with this solution.

I have since found numdiff, which was inspired by ndiff. It looks as though it addresses the problems I've seen with ndiff, so it would be good to try that alternative.

Am closing this issue. Will open a new (low priority) issue to test the use of numdiff.