Allow CLI to retry processing until no new messages are exported

stefpiatek commented 2 months ago

Definition of Done / Acceptance Criteria

As someone exporting data from PIXL, I'd like to be able to set a run off that will deal with the VNA being flaky at returning imaging studies, and it will keep on retrying unexported images until a round of processing doesn't export any images

Testing

could use this for one of the system tests to ensure that it waits until rabbitmq is empty, but with a shorter timeout for checking whether there has been a change in exported images. Other option is to mock out the interaction with database, queue and time to be able to control this precisely and have quick tests

Documentation

Create documentation for use

Details and Comments

Currently have to do this manually to compensate for the VNA, to serve projects easily we should automate this.

The VNA has to pull data from long term storage to a cache layer and often fails during our request for the image. If there's a retry within 24 hours, it then works because the data has been transferred to the cache. Currently the pipline is run a few times over. As we filter out exported images before publishing messages to rabbitmq, requerying is reasonable.

Basic thoughts on how this might work

define last_exported_count, number of messages exported at the start of the run. This would be 0 if the batch is new, otherwise queried from the database.
populate messages
poll rabbitmq until the rabbitmq imaging queue is empty
wait 5 minutes to allow for orthanc raw and orthanc anon to process and the image to be published
query database for the current number of messages exported for the project. If this is the same as last_exported_count, success and exit! If not then query again

Might be worth setting a CLI argument for maximum number of retries, maybe 5?

peshence commented 1 month ago

in progress work at https://github.com/UCLH-Foundry/PIXL/tree/peshence/cli-retry-processing in case whoever picks this up finds it useful (likely they won't, sorry)

stefpiatek commented 1 month ago

It was useful 🎉 quick bit of tweaking and I think we're all set

UCLH-Foundry / PIXL