Refactor existing "Pull" and "Convert" code to use Jobs

[alex-ketch]

Currently the Pull and Convert functions block the main thread and degrade the server performance. To fix this the functions should be converted to use the new Jobs mechanism.

• [x] Write a Task which calls out to Encoda for Convert
• Pull behaviour needs to be standardized across services:
  • [x] Google Docs
  • [x] GitHub
  • [x] HTTP
  • [x] eLife
  • [x] PLOS

---

How to provide feedback to user: Consideration needs to be taken around communicating conversion progress in the UI.

Option 1 Keep current UI, the conversion button spinner will periodically check for the completion status of the now asynchronous Convert Job

Option 2 UI is optimistically updated, showing the file to be outputted in the Files list:

• Conversion Job is spawned in the background
• The file is shown with a spinner until the Job is completed. On success show a temporary checkmark which disappears (when?). on failure show a cross, and a "See Errors" text linking to the Jobs/Activity log

This second option, while likely a better UX, it is more involved and would require changes to the Snapshotting and likely the Project file mounting behaviours. For the time being, we should go with the first option.

[nokome]

An update:

• the pull_elife function is implemented in https://github.com/stencila/hub/pull/436 - some minor changes to be made there but should be merged soon.

@richardmansfield could you next work on:

• pull_plos function (it is very similar to pull_elife and the Typescript implementation is here)
• pull_http function: move over functionality in the exiting fetch_url (and other nearby) functions:
  • custom User-Agent header (I think this avoids issues with some providers if using requests default?; currently a setting but I don't see why it can be a hard coded value
  • if is_malicious_host: raise an exception
  • etc

I'm going to work on integration in the director. My current ideas for that are to add some API endpoints, primarily for testing purposes:

• a /projects/{id}/sources viewset to add a new Source model to a project
• a /projects/{id}/sources/pull action on that viewset to pull all the sources by creating jobs for them and sending them to Job queue.
• add ElifeSource and PlosSource models which implement the pull method by creating a Job model that gets sent to the job queue

[alex-ketch]

The foundational work necessary for this issue were done by @richardmansfield in the following PRs:

• 436

• 438

• 443

• 452

• 469

The second part of the original issue, using the Jobs mechanism from the Hub UI, remains to be done

[nokome]

Done