Implement testing in GitHub Actions

[nikosavola]

Is your feature request related to a problem? Please describe.

Tests are not run automatically on commits, or tags/versions for that matter. This would better see if bugs are introduced.

Describe the solution you'd like

Implement GitHub Actions workflow compiling from source, and subsequently running the tests. This should be possible for Ubuntu, macOS, and Windows.

[nikosavola]

I'm a bit confused about the tests, do they currently require some previous version to compare to? Additionally, is it enough to compile everything with:

cd src
make
make seq OPTIONS=SPARSE
make mpi OPTIONS=SPARSE

and then run the tests like:

cd tests/2exec
test_all cross all

[myurkin]

Niko, thanks for creating this issue. We have been thinking about it for some time, the related issues are #6, #114, #282 . To sum up, here are the main concerns:

• automatic compilation - by default ADDA needs external packages that need to be set up, so the simple commands that you mentioned may be not enough. As discussed in CompilingADDA, there are many existing options to compile ADDA and there are no configure script (#114), although the Makefile provides similar functionality. Still quick compilation recipes may work out of the box in GitHub actions (I am not aware if anybody tried it).
• for compiling use devtools/build - it will do the same as you wrote, but additionally add _spa to sparse binaries to use them together with main ones. You can also run devtools/test_new - it will build and test everything (very extensively).
• currently the most extensive tests are indeed tests/2exec, but they are designed to compare two binaries (of different versions) or different modes. test_all cross all (that you mentioned) will run without a previous version, but it will not catch many possible bugs (that, e.g., change the result in both seq and mpi mode). Comparing against previous binary is the most sensitive one and it can run out of the box on Windows (due to available binaries in win64/). For other OS, some automatic compilation of the last stable version can probably be implemented.
• current tests (for instance, test_new is too extensive, can take hours on my laptop), so I run it only before a release. For simple changes I usually do something like tests/comp2exec seq unless I have some concerns about other modes. So probably we need to differentiate them in GitHub Actions as well. It also should be possible to accelerate the tests by reducing the default grids (for instance, from -grid 8 to -grid 4) but that should be tested not to produce any unexpected differences.
• also we have a few simple tests in tests/equiv, which are designed to use a single ADDA binary running it with different (but presumably equivalent) command lines. But they currently include only the tests for superellipsoid shape and Bessel beams and are not ready for use in automatic scripts.

To conclude, it would be great if you can make any working prototype of GitHub Actions. And let me know, if you have further questions or ideas.

[myurkin]

And I think that any simple test is better than none. So the simplest would be

<install some packages, at least fftw>
cd src/
make seq
cd ../tests/2exec
./comp2exec seq

but the problem with previous binary remains.