The code uses the requests Python package to send HTTP requests.
The Python requests library needs to load file content in memory
when the files argument of the post function is used. In our systems,
we think this is the cause of some hangs.
One possible workaround is to use requests-toolbelt which provide a
class to generate the request body of the HTTP POST query. With
requests-toolbelt, the multipart-encoded data is injected
using the data parameter.
With this change, the upload command does not hang. Further, the memory usage
is now constant.
Command tested:
$ arast upload --single SRR1061345_1.fastq
The md5 checksum is identical in the file system on the client and in
the object storage file system.
The code uses the requests Python package to send HTTP requests. The Python requests library needs to load file content in memory when the files argument of the post function is used. In our systems, we think this is the cause of some hangs.
One possible workaround is to use requests-toolbelt which provide a class to generate the request body of the HTTP POST query. With requests-toolbelt, the multipart-encoded data is injected using the data parameter.
With this change, the upload command does not hang. Further, the memory usage is now constant.
Command tested:
$ arast upload --single SRR1061345_1.fastq
The md5 checksum is identical in the file system on the client and in the object storage file system.
$ md5sum SRR1061345_1.fastq f13b4842984fccf7324a6b82c34baa1b SRR1061345_1.fastq
The date makes sense too, see below.
Finally, I verified that the owner is correct in the object in the object storage system.
Signed-off-by: Sébastien Boisvert boisvert@anl.gov Link: https://github.com/kbase/assembly/issues/167