We should set up automatic regression testing. Specifically, we need to add a test that upgrading the pipeline and/or backend will not break things (unless that is expected). This is what I did manually for #390:
Get a backend and a pipeline (with concurrency 1) running with the old code.
Start a job through the old backend.
Queue a job for the old pipeline through the old backend.
Queue a job for the new pipeline through the old backend.
Upgrade the backend, i.e. pull the code, and restart everything but the pipeline.
Queue a job for the old pipeline through the new backend.
Queue a job for the new pipeline through the new backend.
Check that the running job from step 2 is still running properly, and that its settings can be adjusted. Abort it or otherwise end it, then do the same for the jobs from step 3 and 6.
Upgrade the pipeline.
Check that the jobs from step 4 and 7 are running properly.
For a single PR, that's fine manually, but there will certainly be more changes like this in the future, especially if we ever decide to move away from this Redis polling model to a proper API (cf. comments on #290) and want to do so smoothly.
This is still not a very thorough test of course. At least we don't need to support running a new pipeline against an old backend since we can always upgrade the latter before the former. But any further ideas of what should be included in such a test are highly welcome.
We should set up automatic regression testing. Specifically, we need to add a test that upgrading the pipeline and/or backend will not break things (unless that is expected). This is what I did manually for #390:
For a single PR, that's fine manually, but there will certainly be more changes like this in the future, especially if we ever decide to move away from this Redis polling model to a proper API (cf. comments on #290) and want to do so smoothly.
This is still not a very thorough test of course. At least we don't need to support running a new pipeline against an old backend since we can always upgrade the latter before the former. But any further ideas of what should be included in such a test are highly welcome.