search

irods / irods_resource_plugin_s3

S3-compatible storage resource plugin for iRODS
Other
12 stars 16 forks source link

Streaming mode - Small files fail when uploading with iput -N2 #2197

Closed JustinKyleJames closed 3 months ago

JustinKyleJames commented 3 months ago

If you upload files with the -N flag to > 1 and the file is small enough that this will require a local cache file to be used (due to minimum multipart part size), in some cases it appears that iRODS is doing the writing with only one thread despite the -N flag setting.

Because of this, the S3 plugin is waiting for other threads to write/close before flushing to S3. Instead iRODS is calling the stat operation after the first file closes. This stat is failing because the file has not yet been flushed to S3.

This seems to be a change between 4.2 and 4.3.

trel commented 3 months ago

yes, please confirm if this is different from #2198

JustinKyleJames commented 3 months ago

I have confirmed that this is the same as #2198. I am closing this issue.