Upload to sharepoint

[chrisflatley]

What happened?

The 3Gb package fails with 429 in crafty-amulet...

Some logs:

raceback (most recent call last):
    Uploaded '1008000000' bytes from '2984138417'...[33.78%]
  File "/home/runner/.local/lib/python3.8/site-packages/office365/runtime/client_request.py", line 94, in execute_query
    Uploaded '1009000000' bytes from '2984138417'...[33.81%]
    Uploaded '1010000000' bytes from '2984138417'...[33.85%]
    Uploaded '1011000000' bytes from '2984138417'...[33.88%]
    response.raise_for_status()
    Uploaded '1012000000' bytes from '2984138417'...[33.91%]
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    Uploaded '1013000000' bytes from '2984138417'...[33.95%]
    raise HTTPError(http_error_msg, response=self)
    Uploaded '1014000000' bytes from '2984138417'...[33.98%]
requests.exceptions.HTTPError: 429 Client Error:  for url: https://committedio.sharepoint.com/sites/External/_api/Web/GetFileByServerRelativeUrl('%2Fsites%2FExternal%2FShared%20Documents%2FIncoming%2F20211017181019%2Fca-deployment-2021-10-17-18-28.zip')/continueUpload(uploadID='471107df-3d50-4883-85a5-c6bea2a397c4',fileOffset=1121000000)

What did you expect to happen?

File should have uploaded.

Any relevant technical information (OS, versions, logs)?

Running in devk8s, die after exactly 1 hour though.

[chrisflatley]

Unclear the issue here:

Potential solutions options

• Does the Python client respect the rate limiting? (Unclear... but I guess we want to avoid getting there anyway).

• Can we use a larger chunk size and upload quicker? I think it's 1mb at the min which is tiny. Could it be bigger (I think I tried and the answer was no at the time). Little information on the actual limits here but perhaps a start point https://techcommunity.microsoft.com/t5/microsoft-sharepoint-blog/always-use-file-chunking-to-upload-files-gt-250-mb-to-sharepoint/ba-p/509988

• Is it actually rate limited, or is it the our devk8s runner? It seems suspicious it dies after 1 hour when in fact the job is 120 mins. Perhaps the runner is being killed and the 429 client errors are merely logging, but being dealt with. Check the k8s logs.

• This is 3gb zip, should it be divided to smaller pieces? Would that work better? (or would be just 1Gb done and then an error).

• Option to upload to OneDrive instead, the API looks the same - so I assume the limitations are the same too.