SOW: Asynchronous Import of URLs

[jillpe]

Summary

SoftServ will move importing of files that come from cloud resources to be asynchronous of the initial processing of the CSV row via Bulkrax. This will likely involve wrapping the original import job with a background job that first fetches the remote resource then begins importing.

Testing Instructions

• [ ] Create a CSV Bulkrax Importer
• [ ] Use this CSV browse_everything_test.csv
• [ ] Click the Attach Cloud Files button
• [ ] Choose the file largetfile.txt
• [ ] Click Import
• [ ] Observe that you were immediately redirected to the Importers index page
• [ ] After some time, confirm that the file work was created and the file was ingested

[jillpe]

SoftServ QA: ✅ Work ingested and file is able to be downloaded Importer

Screenshots

 

• [x] Create a CSV Bulkrax Importer
• [x] Use this CSV browse_everything_test.csv
• [x] Click the Attach Cloud Files button
• [x] Choose the file largetfile.txt
• [x] Click Import
• [x] Observe that you were immediately redirected to the Importers index page
• [x] After some time, confirm that the file work was created and the file was ingested

[NoraRamsey]

BL QA: ✅ Work ingested and file is able to be downloaded Importer

• [x] - [ ] Create a CSV Bulkrax Importer
• [x] Use this CSV browse_everything_test.csv
• [x] Click the Attach Cloud Files button
• [x] Choose the file largetfile.txt
• [x] Click Import
• [x] Observe that you were immediately redirected to the Importers index page
• [x] After some time, confirm that the file work was created and the file was ingested

[grahamjevon]

Will continue testing once Rory has got AWS buckets ready.

[cziaarm]

Confirmed (by me) that bulkrax import of large file from browse-everything s3 was successful: https://bl.bl-staging.notch8.cloud/concern/articles/2f1325f5-f962-4279-b83b-6dc22ef259db?locale=en

[j-basford]

@NoraRamsey can you test this on Thursday please?

[ShanaLMoore]

@cziaarm Can we close this ticket? If not, what is remaining? cc @kirkkwang

[cziaarm]

@cziaarm Can we close this ticket? If not, what is remaining? cc @kirkkwang

Hi @ShanaLMoore nothing is remaining on this ticket, but because the BL are testing with larger files and these are failing due to (i think) modeshape version in use by fcrepo https://assaydepot.slack.com/archives/C0313NK2LJ0/p1712662620970479?thread_ts=1712156637.197069&cid=C0313NK2LJ0)

I'll make a new ticket for that (#534) and then this one can be moved along