Goal:
Have a pipeline establish that executes the required jobs that implement the archiving functionality and notifications of job and dataset status updates shall be forwarded reliably (i.e. retries) to SciCat.
Any actual interactions with the dataset shall only be mocked for testing
Tasks:
[x] Add sequence of task flow
[x] Verify SciCat interactions and request data with #41
[x] Verify correct error handling and updates of SciCat jobs and dataset lifecycle
~- [ ] Add logging and notifications of task failures~
[x] Handle multiple datasets in same job
Smaller tasks:
[ ] ~Add worker pool to job system~ moved to different milestone
[x] Add proper config parsing
[x] Add table of job and dataset lifecycle: blocked
[x] add fetch of origdatablocks
[x] handle return of scicat requests properly
[x] add retry handling of scicat request
[ ] add cleanup step for staging at the end
[x] split config of backend and worker
Definition of done:
Test suite is passing with all the defined interactions implemented.
phwissmann
commented
3 months ago
Goal: Have a pipeline establish that executes the required jobs that implement the archiving functionality and notifications of job and dataset status updates shall be forwarded reliably (i.e. retries) to SciCat. Any actual interactions with the dataset shall only be mocked for testing
Tasks:
Definition of done: Test suite is passing with all the defined interactions implemented.