radiocosmology / alpenhorn

Alpenhorn is a service for managing an archive of scientific data.
MIT License
2 stars 1 forks source link

Asynchronous transfers with a task queue and Nearline support #137

Closed james-s-willis closed 1 year ago

james-s-willis commented 3 years ago

@jrs65 @ketiltrout I've added the nearline test, so this is ready to review now

anjakefala commented 3 years ago

From a conversation with James.

This PR has the following goals:

task.py contains the new Task logic. update.py has been then factored out, and can now use add_task(TaskClass) for every type of Task type.

The build is currently red because some tests check for help output, and one Python package has been updated since the code was written, changing the output. It still needs to be decided whether to fix the version of the Python package or update the tests.

ketiltrout commented 3 years ago
  • To begin bringing over fixes that have been added to alpenhorn 1, but have not been ported to alpenhorn 2

Unless there's specific things needed to support the task queue, I would do this part in a separate PR.

ketiltrout commented 1 year ago

Whilst the functional parts of James's work is still present, I've largely rewritten the task/queue framework.

I'll going to create a new PR for that.