search

artefactual / automation-tools

Tools to aid automation of Archivematica and AtoM.
GNU Affero General Public License v3.0
46 stars 33 forks source link

Add transfer_async #53

Closed sevein closed 6 years ago

sevein commented 6 years ago

transfer_async.py is an entry point that reuses what's in transfer.py but hijacks the only function that needed to be changed (start_transfer) so the script works against the new /api/v2beta/package API.

transfer_async.py does not run pre-transfer scripts because the transfers are started automatically so there is no room for that. In the future, /api/v2beta/package may learn a new open status that may allow us to solve this limitation.

This can be considered a walking skeleton. The plan is to eventually have it replace transfer.py. The advantage is that /api/v2beta/package does not block. This was a priority @ JiscRDSS because you can't have a connection open for hours like we do when we're copying big transfers - /api/v2beta/package solves that problem.

I've been testing in my Compose environment and the following command:

python -m transfers.transfer_async \
    --log-level DEBUG \
    --user test --api-key test --am-url http://127.0.0.1:62080 \
    --ss-user test --ss-api-key test --ss-url http://127.0.0.1:62081 \
    --transfer-source "b002cea4-82cf-4bc5-b4e5-fbc87443540d" \
    --transfer-path /home/test \
    --transfer-type standard

Issue: https://jiscdev.atlassian.net/browse/RDSSARK-480. It depends on https://github.com/artefactual/archivematica/pull/936.

sevein commented 6 years ago

@ross-spencer maybe you can take a quick look to this PR? It adds a new experimental transfer_async.py that we're going to be using @ JiscRDSS so it doesn't really change existing workflows. transfer_async.py may be evolving as we update the /api/v2beta/package API.

ross-spencer commented 6 years ago

@sevein Acknowledging I still need to look at this! - One branch relevant to this is, https://github.com/artefactual/automation-tools/commit/76a8b61b5267eed234fd5ce72ff52f3dcf335232 (utils.py is modified) I think we should get this reviewed and merged and then rebase your PR here to fit - what do you think? I'll check with @hakamine if he has any more info on issue #51 but I think this change is independent of that, whatever is causing #51 can be handled in another branch.

sevein commented 6 years ago

I think we should get this reviewed and merged and then rebase your PR here to fit - what do you think?

Sure. Ideally we wouldn't block paying work with minor stuff but I can understand the situation.

sevein commented 6 years ago

Rebased.

ross-spencer commented 6 years ago

On my task list for the end of the day! :+1: