pwilloughby
commented
1 year ago It looks like the key length is validated and rejected with a 400 and the correct error KeyTooLongError. https://github.com/storj/gateway-st/blob/main/miniogw/gateway.go#L809. The problem appears to be slashes explode the encrypted length. The key a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a is less than 1024 bytes but the encrypted length is very long
uplink: metaclient: key length is too big, got 16202, maximum allowed is 1280
halkyon
storj-gerrit[bot]
We noticed errors like "uplink: stream: metaclient: key length is too big, got 1451, maximum allowed is 1280" showing up in the gateway logs often. It's not mapped to any S3 error response, so it defaults to 500 returned to the client (instead of something like a 400), which isn't great.
Maximum key length described by S3 is 1024 bytes: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html "The object key name is a sequence of Unicode characters with UTF-8 encoding of up to 1,024 bytes long.".
Metainfo endpoint is ensuring a maximum length of 1280 of the encrypted key length, since we do not send the unencrypted key from uplink. Here's where the error is coming from: https://github.com/storj/storj/blob/25f2305e0078bbe96c1a87668b9d436797aba1e4/satellite/metainfo/endpoint_object.go#L77
~I would say the solution is to validate the unencrypted key is no longer than 1024 bytes in gateway and reject the request earlier instead of sending it to metainfo to fail. We already have validation in gateway we can expand upon: https://github.com/storj/gateway-st/blob/main/miniogw/validate.go. I believe we should map invalid key lengths to the error type
minio.ObjectNameInvalidunless there's a more specific error type.~We currently validate the object key length here: https://github.com/storj/gateway-st/blob/main/miniogw/gateway.go#L809 but it appears that some keys, e.g. those with many slashes
a/a/a/a/a/a/a/a/a/a/a/ ...up to 1024 bytes are accepted but end up exploding in length over 1280 bytes when the key gets encrypted.