Closed jetmind closed 1 year ago
Thanks for the repro!
As I assume you've surmised, the problem comes from aws-api reading the InputStream into a byte[] before io/input-stream
has consumed all the bytes in f
. There is no error for AWS to report because it's valid to :PutObject
with an empty value, so there's nothing for us to do there.
We could try to ensure we read all the bytes, but I modified the code in aws-api to use .readAllBytes
and this problem persisted, so I'm not quite sure how we could do it.
Thanks for prompt response!
Yeah, that was my thinking but I didn't manage to reproduce it without actually writing to S3, so something more intricate might be at play here.
One of the things I wanted to look at is how underlying http client handles the body but it seems like it's code isn't public?..
Anyway, revisiting it again I think I know what the problem is. See, util/->bbuf
is called inside the send-request
, which could be retried and will try to read the same input stream again, resulting in empty ByteBuffer
.
One of the things I wanted to look at is how underlying http client handles the body but it seems like it's code isn't public?
It is, but not in the way everybody has grown accustomed: the source is in plain text in the jar. It's just not conveniently posted on a site like github.
Anyway, revisiting it again I think I know what the problem is. See,
util/->bbuf
is called inside thesend-request
, which could be retried and will try to read the same input stream again, resulting in emptyByteBuffer
.
Brilliant! I didn't think of that as I was perusing. I'll look into getting that fixed.
Fixed in https://github.com/cognitect-labs/aws-api/commit/b6c861d14a3cce67258828a9ce2f0ff50c45194e. I'll close this when we release. Should be some time this week or early next.
Released in v 0.8.652
Dependencies
Description with failing test case
Concurrent writes to the same S3 key could leave invalid (empty) objects without reporting any errors to the caller. Seems to happen only if body passed to :PutObject is an InputStream.
Here's how to reproduce:
Run it and you'll get something like this (re-run a few times if it didn't fail, but usually it does):
Note the different
ETag
in the last response and:ContentLength 0, :Body nil
in the:GetObject
response.Passing byte array as a :Body seems to fix the issue. Note
.readAllBytes
to convert input-stream to the byte array first.Re-run:
Object now has proper
:ContentLength
and body is there.