search

VERITAS-Observatory / V2DL3

VERITAS (VEGAS and Eventdisplay) to DL3 Converter
BSD 3-Clause "New" or "Revised" License
8 stars 1 forks source link

v2dl3-vegas CI #153

Closed tmcahill closed 2 years ago

tmcahill commented 2 years ago

This update brings v2dl3-vegas CI testing, as well as some useful utilities for users.

Minor changes:

CI Testing

Pulls a Docker image that I've set up with all of the prerequisite software installed for v2dl3-vegas CI testing. Attempts to install the new branch, then generates fits outputs for a variety of inputs and flags. This is then repeated for the main branch, and finally all fits files are compared via fitsdiff.

Since fits are regenerated from main each time, when intentional changes are desired, simply check the failed fitsdiff test to make sure only the intended changes are detected, then merge the branch. The following CIs will generate from the new main branch with the incorporated changes.

The Docker image will print all standard output to the test log so that test failures are easily diagnosable. The recipe for the image is included in the update. The test inputs I have baked into the image after building from the recipe, as those must stay private. Examples of a successful test and a test failing due to fits differences are available to view as the open pull requests for this fork. https://github.com/tmcahill/V2DL3-VEGAS-CICD-Test/pulls

I've written the CI such that any inputs may be provided should they need updated or replaced in the future. Just provide the effective area filepaths and a directory containing stage5 files to utils/vegas_docker_test_runs.sh

The image itself is hosted on a private Docker Hub repo as it contains VEGAS and the test inputs which cannot be exposed. The Docker login credentials are to be stored as Github repository secrets.

It may be better to host the image and inputs somewhere ourselves instead, then store the wget link as a repository secret. I can implement this change if you have somewhere to host these. The image is ~6 GB and the inputs ~250 MB

Utilities

After adding lots of new utility files, I thought it efficient to organize all utility files into a 'utils' folder. I've updated and re-tested the ED CI paths accordingly.

I wrote a separate README for the Docker stuff in its directory. The runlister explains itself in its help string. References to both of those have been added to the main README.

The new shell scripts are used by v2dl3-vegas CI.

GernotMaier commented 2 years ago

Closing this and move the discussion to PR #154