upload client

[mikeAdamss]

What is this

In order to upload files to the dp-upload-service we need to present files in chunks wirh very specific fields to enable uploading large files in peices (chunking) and in a resumable way.

What to do

The upload service has a swagger spec that you can put into a swagger editor here: https://editor.swagger.io/ . there's something not quite right about the spec but it'll work enough that you can see some endpoint documentation.

Jim Bryant (DE team) has already written some code to do this and it works ...but... I believe its working with the old version of the upload service (/upload rather than /upload-new which we want.

It does however have all the chunking in it as well as a lot if not all of the logic.

• https://github.com/ONS-OpenData/cmd-run-transform/blob/9b164dd758c01f6eed4d1bb7cbd4836083943007/clients.py#L1112
• https://github.com/ONS-OpenData/cmd-run-transform/blob/master/clients.py#L1161

We just need to:

• wrap it in a class to become an upload client
• add unit tests, split out the logic a bit to make it easier to test.

Do use the BaseHttpClient as that'll give you exponential retries which you're gonna want.

So it'll be something along the lines of:

class UploadClient(BaseHttpClient):

    def upload_new(self ... whatever you'll need here):
        # encapsulate some of the logic from Jims scripts to work with /upload-new

Once you think its there or thereabout and you need to sanity check you'll need access to the staging environment (as there's an upload service there), but grab @JimBryant7 to help with that please, we should be able to point you at where any test files you pass through appear.

Staging isnt bleed, we'll need to upload things to check this works but tidy up after yourself please.

Have a look at vcr for the unit tests, but be careful that there's no credential information getting recorded (look at the yaml "casette", its fairly plan english). There shouldnt be but we really dont want to be proved wrong - confirm this before you commit any casettes to git.

Acceptance Criteria

• [ ] Upload script transcribed to upload client
• [ ] Unit tests

[SarahJohnsonONS]

Blocked due to issues with accessing the DP Upload Service. Current changes on branch https://github.com/ONSdigital/dp-python-tools/tree/%2310-upload-client.