Is your feature request related to a problem? Please describe.
A user wants to upload a large file, primarily using a commandline client or API to the repository.
The current process is using the file upload API, which:
is slow for large files, as the file is uploaded in a single thread/request
it is prone to network errors and whenever this occurs, the user has to re-upload the whole file
Describe the solution you'd like
Invenio provides an API/client to enable uploading chunks of large data. At the beginning of the upload,
the client/user will determine the number of chunks. Invenio will provide a URL for each of those chunks
which client/user will use to perform a PUT request to upload the content of the chunk.
Describe alternatives you've considered
The file might already be present on the target storage - in this case, a linking should be used instead
of re-transfer.
Is your feature request related to a problem? Please describe.
A user wants to upload a large file, primarily using a commandline client or API to the repository. The current process is using the file upload API, which:
Describe the solution you'd like
Invenio provides an API/client to enable uploading chunks of large data. At the beginning of the upload, the client/user will determine the number of chunks. Invenio will provide a URL for each of those chunks which client/user will use to perform a PUT request to upload the content of the chunk.
Describe alternatives you've considered
The file might already be present on the target storage - in this case, a linking should be used instead of re-transfer.
Additional context
See the S3 multipart upload documentation at https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html for details how it is done in S3 protocol.