search

aws / aws-sdk-cpp

AWS SDK for C++
Apache License 2.0
1.93k stars 1.05k forks source link

AWS SDK can't handle unusual object keys #1924

Open crusader-mike opened 2 years ago

crusader-mike commented 2 years ago

Describe the bug

There are multiple issues with SDK that results in it's inability to handle unusual object keys (even if they are supported by underlying platform, be that AWS S3 or anything else). Examples of object keys:

a//
a///b
a/./b
a/..
and etc
  1. Issue with repeating slashes is caused by this line -- StringUtils::Split() discards empty elements. This in turn causes us to access completely different object (e.g. a/b instead of a///b) with a whole shebang of possible results -- from race conditions to data loss.
  2. Apparently, most systems treat path component of URI in a special way -- before it gets processed path gets split into components (using / for separator) and if said component is . or .. an attempt is made to get rid of it. After this URI gets used in downstream logic (which checks signature, etc). Naturally, this causes Signature Mismatch errors on server side because it observes slightly different URI than one sent by client. All because this SDK decided not to percent-encode . when building URI. Encoding it manually won't help because then it will be encoded second time by SDK. Basically, if your application tries to access an object with key a/./b -- it won't be able to.

Such object keys are relatively rare, but they happen. Not every application out there is written using AWS SDK. Some of them implemented this stuff properly and create objects this SDK can't deal with. This submission is result of couple days of investigations on "why my ECS -> S3 migration fails?" topic at major organization.

Bug is related somewhat to #1560 (with bugs in ECS implementation of V4 signature calcs it is even more convoluted)...

Expected Behavior

I expect SDK to work regardless how weird/unusual symbols that comprise object key are.

Current Behavior

It doesn't work. My app ends up hitting wrong objects, failing with no such object, HTTP codes 444, 404 and even 500 depending on storage device it is used against.

Reproduction Steps

Should be pretty obvious from description.

Possible Solution

  1. Do not discard empty components in URI::URLEncodePath and other places
  2. Encode . when building URI that will be used in HTTP request

Additional Information/Context

No response

AWS CPP SDK version used

1.7.116

Compiler and Version used

VS2019/GCC

Operating System and version

Win10, Linux

jmklix commented 1 month ago

We have made some improvements to this sdk in how it handles unusual object keys. This includes a security fix. I did test your short list and some of them currently work, while others don't. Both a// and a///b upload a file, while a/./b and a/.. fail with 

Error: putObject: The request signature we calculated does not match the signature you provided. Check your key and signing method.

As noted in the s3 user guide here, some characters (ie. forward slash "/") might require special handling. Please let us know if you have any questions about how this sdk handles special object keys.

crusader-mike commented 1 month ago

Proud of you, guys... 2.5 years after I figured out all details you've implemented smth that covers a subset of reported cases. It doesn't really help me -- simply because I had to hack this SDK (intercept certain things) to force it to behave correctly. As a side effect now it is pretty darn hard to upgrade SDK version in the product (I was working on). Effectively, I had to freeze the SDK version indefinitely. But on the bright side -- it handles all these weird cases correctly. :D Including bugs in ECS implementation of signature calcs.