build: Test Models Against the Full Docker Stack

allgandalf commented 1 year ago

Description

In this PR, we have used the full docker stack to test SIPNET by using Docker Compose, this test will run weekly on Mondays.

Motivation and Context

From Discussions in #3184, It would be wise to have a script file and have our cURL POST requests for different models in it as the Matrix option would have to bring up the whole docker stack for each job of the Matrix.

In this PR, we have added the SIPNET model test, which outputs the sipnet.out (model output) file, Test results can be found in this workflow

In the script file, template has been provided to correctly input data into the cURL request so that in future, People can add more models in the script file against which tests can be performed.

Review Time Estimate

[ ] Immediately
[ ] Within one week
[x] When possible

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My change requires a change to the documentation.
[ ] My name is in the list of CITATION.cff
[ ] I have updated the CHANGELOG.md.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

This PR closes #3184

infotroph commented 4 months ago

Left some comments above focused on technical implementation, but from a broader perspective it's worth making sure we're aligned that "Whole-docker-stack Sipnet test via GitHub Actions every Monday" is a good approach. For my part I agree it's smart to make sure we have tests that exercise the whole docker-compose project, but given we already test Sipnet in the daily CI suite I don't have a good sense how much more information we get from testing it again inside the full Docker stack. Emphasis that I really do mean "I don't know", not "I think the proposed approach is wrong"

Pinging @meetagrawal09 for thoughts on how this aligns/interacts with your planned CI development this summer

meetagrawal09 commented 4 months ago

@infotroph Some points that come to my mind when I check this implementation are :

Should we be more selective as to which containers we are running as part of this full docker stack rather than running all of them at once.
We are restricting ourselves with the features provided by the PEcAn API by using curl queries to trigger model runs. A more extensible approach seems to be by using a settings file and triggering workflows by running the workflow.R script.
We can run multiple models here as part of the script in the same job but I prefer having separate models to be running as separate jobs. E.g. Here I can run SIPNET and ED2 as part of the shell script but they both are part of the same job. In case SIPNET fails to run correctly this entire job fails without even trying if ED2 was able to run or not. Rather if I configure them as 2 separate jobs we can analyse issues with each model separately. By creating a new job for each model run we are separating out concerns with their failures and also reducing time on each run. It also gives us the liberty to choose which model we want to trigger a run for as part of the CI.

allgandalf commented 1 month ago

Closing this PR in favour of https://github.com/PecanProject/pecan/pull/3345, please reopen if you disagree

PecanProject / pecan