Decision letter parsing workflow

[gnott]

To be done is to create a workflow and activities to process decision letter parser .zip files as they are added to a bucket.

• [x] First step is to fill out this issue with some tasks with checkboxes

Summary

This issue describes an implementation of processing decision letter & author response content. The majority of the parsing logic is held in the decision-letter-parser code library. The elife-bot can be used to define and run the steps of the processing workflow.

A .zip file, which contains a .docx file and optionally some figure images and/or video files is copied to an S3 bucket. An S3 notification from that bucket is read by a queue listener in the elife-bot. As a result, the bot will start a workflow. That workflow will have activities run in series to validate the input, generate JATS output from the .docx file contents, copy file assets to another S3 bucket, POST the JATS output to an API endpoint. This workflow will notify people if it encounters an error, or when everything completed successful, it will notify people of the completed status.

Bot workflow and activity build out in brief:

Workflow parts to build

• [x] Create workflow: name IngestDecisionLetter
• [x] Create a workflow starter for IngestDecisionLetter
• [x] Add code tests for the workflow and starter
• [x] Start an IngestDecisionLetter workflow execution when a file is uploaded to the input bucket

Activities in this workflow to create

(separate issues can accompany each for more detailed discussion)

• [x] Note: will probably want to create a new provider, named letterparser.py perhaps, that can store any code to be shared by these activities, such as logic to unzip the contents, build from .docx files, and such
• [x] ValidateDecisionLetterInput (https://github.com/elifesciences/elife-bot/issues/966)
• [x] GenerateDecisionLetterJATS (https://github.com/elifesciences/elife-bot/issues/967)
• [x] DepositDecisionLetterIngestAssets (https://github.com/elifesciences/elife-bot/issues/968)
• [x] PostDecisionLetterJATS (https://github.com/elifesciences/elife-bot/issues/969)
• [x] DecisionLetterReceipt (https://github.com/elifesciences/elife-bot/issues/970)

External infrastructure required for completion

• [x] S3 bucket for decision letter input files (https://github.com/elifesciences/issues/issues/5146)
• [x] S3 bucket to store decision letter parser assets output (https://github.com/elifesciences/issues/issues/5109)
• [x] API endpoint to receive decision letter JATS XML (to be provided by ExeterPremedia)

Assistance required from infrastructure team

• [x] Add configuration file (letterparser.cfg) that specifies the eLife DOI and file naming structure
• [x] Ensure Docker is running and can run docker image for invoking the pandoc utility (https://github.com/elifesciences/issues/issues/5194)

Once built we can

• [x] Test the workflow on the continuumtest environment
• [x] Deploy to the prod environment and start using it

More tasks can be added to this issue as they are discovered or are defined in more detail.

[gnott]

Related to https://github.com/elifesciences/decision-letter-parser/issues/43, where the bucket and S3 notifications were connected, from which this workflow will be triggered.

[Melissa37]

Thanks @gnott for creating this ticket.

Can you indicate which of the items to be ticked off that you cannot do without the internal eLife team?

[gnott]

I believe only https://github.com/elifesciences/issues/issues/5194 will be the highest priority item for eLife Internal - thanks @Melissa37 👍

[gnott]

This is feature complete now. I expect we'll do some improvements over time once people start using the decision letter ingest workflow.