AllanEngland
commented
7 months ago Ok, so I've made significant changes to how the import works, mostly on the API side of this branch.
An imported proposal is mapped to an ObservingProposal and parsed with any new organisations and people added to the database, then updateClonedReferences() is called before the new proposal is persisted. Old links to supporting documents are pruned, following this each document in the uploaded zip file is added as a supporting document. The uploads happen in their own transactions, so an individual one can fail (hopefully with an error reported!) without the whole import being aborted.
There are still some outstanding quirks:
- Export produces a pdf of the proposal overview pane, this is quite useful, but on import it's uploaded as a supporting document. This doesn't happen to proposal.json because it has a specific name that's filtered out, perhaps the overview should also have a specific filename.
- The API has a file upload maximum size of 20MB, this is somewhat restrictive for supporting documents. I'll raise an issue on the API project to deal with this, however, once done the GUI constant
MAX_SUPPORTING_DOCUMENT_SIZEshould be updated accordingly. - An unmodified export zip file will import correctly, but if I edit its contents and create a new zip on my Mac, import can no longer find the proposal.json file. Don't know if this is just a MacOS problem and need to investigate further.
this implementation tries to do the entire upload via already created client rest APIs.
Therefore it wont handle REST exceptions very elegantly.
a better approach would be to send 1 rest request, with the proposal JSON and all supporting documents and then using a transaction from the entityManager to do the entire upload, and be able to do an transaction rollback if any errors occur. But as I don't know how to make a REST API with multiple files as well as the JSON in the body. I'll be doing the more clunky process, as I could put the JSON and then do each separate document, but any failure would still flood the system with deletes, so going to go for the full client side implementation.
export component: utilises JSON.stringify to create an json representation of the proposal and place this within the zip (its worth noting documents are already put into the zip from previous work). import component: results in the separate bits having new requests to the database to acquire their database ids and then adjusting the proposal and corresponding objects directly to reflect the new ids. It then finishes with sending the new modified objects back to the database.