search

steveandroulakis / mytardis-lighthouse-migrate-test

testing migrations from http://mytardis.lighthouseapp.com/ to Github Issues
0 stars 0 forks source link

'Import Staging' async feature #39

Open steveandroulakis opened 11 years ago

steveandroulakis commented 11 years ago

Due to remote copying, the 'Import Staging' feature can take quite a while and hold up the web interface in the process. I propose making the action a celery asynchronous task, with email notification on success/failure.

This work was done personally for compatibility with 2.5 here, and should be able to be fashioned for 3.0: https://github.com/steveandroulakis/mytardis/commits/mytardis_monash_deployment

Requires Joanna Huang's filter fix for staging (also listed in the commits).

original LH ticket

This ticket has 0 attachment(s).

steveandroulakis commented 11 years ago

’Import Staging’ async feature

by Steve Androulakis

steveandroulakis commented 11 years ago

’Import Staging’ async feature

The main issue with this is the verification. The background import tasks assume that every job in them is able to be verified against a SHA-512 or MD5 hash.

To generate that hash, you need to read the file. Once you’ve read the file, there’s a very small overhead in actually storing the file locally. So as a result, staging is still synchronous and it skips the background queue.

A potential fix would be making "verify" aware that staging files are safe to ingest without digest checking, but that seems... inelegant. Another option would be to just flip the "verified" flag without checking the file, but that would cause staging files to be accessible before being locally copied. You’d also have to have the background import skip verification for already verified files (which it doesn’t currently do).

by Tim Dettrick

steveandroulakis commented 11 years ago

’Import Staging’ async feature

Written async file staging (with verification included and an email notification)

Needs unit tests for this new method, and the email part (which works off plain text and html templates).

The interface part will come next, including dialog for getting someone’s email address, much like Tim did for the RIF-CS requirement.

by Steve Androulakis

steveandroulakis commented 11 years ago

’Import Staging’ async feature

This is almost done..

by Steve Androulakis