ECP-WarpX / WarpX

WarpX is an advanced electromagnetic & electrostatic Particle-In-Cell code.
https://ecp-warpx.github.io
Other
309 stars 196 forks source link

Integrate the regression tests into the automated CI #855

Open RemiLehe opened 4 years ago

RemiLehe commented 4 years ago

Right now the regression tests are run on the battra machine, every night. However, this causes tests to fail after PRs are merged. Instead it would be good to enforce that PRs pass the refression tests, by integrating them in Travis CI.

This would essentially require the Travis CI instance to download the reference test results, from a dedicated Google Drive. Here are the corresponding action items:

ax3l commented 4 years ago

I checked the APIs for up- and download and GDrive has the problem that we have to use an actual account for uploads (no public writable shared key/url-token possible). Have to check if an extra Google Account is okay for such workflows or if we take another service.

ax3l commented 4 years ago

As discussed today, another option is to publish the build artifacts alongside the HTML summary that is already rsync-ed to a web-server at https://ccse.lbl.gov/pub/

We discussed today if we can also find a workflow to update the regression tests automatically. One way might be to create a text file or directory of text files in the WarpX source that contains a serial-number (increase only) or hash + a list a tests. These would be the regression tests that shall be re-build the next night after a PR was merged that intentionally broke (updated) a few regressions.