Closed alex-ketch closed 4 years ago
An update:
pull_elife
function is implemented in https://github.com/stencila/hub/pull/436 - some minor changes to be made there but should be merged soon.@richardmansfield could you next work on:
pull_plos
function (it is very similar to pull_elife
and the Typescript implementation is here)pull_http
function: move over functionality in the exiting fetch_url
(and other nearby) functions:
User-Agent
header (I think this avoids issues with some providers if using requests
default?; currently a setting but I don't see why it can be a hard coded valueis_malicious_host
: raise an exceptionI'm going to work on integration in the director
. My current ideas for that are to add some API endpoints, primarily for testing purposes:
/projects/{id}/sources
viewset to add a new Source
model to a project/projects/{id}/sources/pull
action on that viewset to pull all the sources by creating jobs for them and sending them to Job queue.ElifeSource
and PlosSource
models which implement the pull
method by creating a Job
model that gets sent to the job queueThe foundational work necessary for this issue were done by @richardmansfield in the following PRs:
The second part of the original issue, using the Jobs mechanism from the Hub UI, remains to be done
Done
Currently the
Pull
andConvert
functions block the main thread and degrade the server performance. To fix this the functions should be converted to use the new Jobs mechanism.Task
which calls out to Encoda forConvert
Pull
behaviour needs to be standardized across services:How to provide feedback to user: Consideration needs to be taken around communicating conversion progress in the UI.
Option 1 Keep current UI, the conversion button spinner will periodically check for the completion status of the now asynchronous Convert Job
Option 2 UI is optimistically updated, showing the file to be outputted in the Files list:
This second option, while likely a better UX, it is more involved and would require changes to the Snapshotting and likely the Project file mounting behaviours. For the time being, we should go with the first option.