uploader fails, returns 400 Bad Request errors

jacob-ablowitz commented 4 years ago

Currently struggling with a nasty problem that has reared its head with both fog-aws and carrierwave-aws, and I'm hoping maybe you know what to do about it. I keep receiving 400 Bad Request errors from AWS; investigation suggests my code is somehow sending the wrong signature - because we're using KMS, we must use v4 signature.

My config/initializers/carrierwave.rb file:

S3_URL_EXPIRATION = ENV.fetch('S3_URL_EXPIRATION', 1.hour).to_i
CarrierWave.configure do |config|

  config.storage    = :aws
  config.aws_bucket = ENV.fetch('S3_BUCKET_NAME')
  config.aws_acl    = 'private'
  config.aws_authenticated_url_expiration = S3_URL_EXPIRATION

  config.aws_attributes = -> { {
    aws_signature_version: 4,
    encryption:            'aws:kms',
    'x-amz-server-side-encryption-aws-kms-key-id': ENV['S3_KMS_KEY_ID']
  } }

  config.aws_credentials = {
    access_key_id:     ENV.fetch('S3_KEY'),
    secret_access_key: ENV.fetch('S3_SECRET'),
    region:            ENV.fetch('S3_REGION','us-east-1'), # Required
    stub_responses:    Rails.env.test? # Optional, avoid hitting S3 actual during tests
  }

end

Extract of AWS CloudTrail event logs (closest I can come to the request at the moment):

userIdentity: {
  type: "AWSAccount",
  principalId: "",
  accountId: "ANONYMOUS_PRINCIPAL"
},
eventTime: "2020-02-06T02:59:01Z",
eventSource: "s3.amazonaws.com",
eventName: "GetObject",
awsRegion: "us-east-1",
sourceIPAddress: REDACTED,
userAgent: "[CarrierWave/1.2.3]",
errorCode: "InvalidArgument",
errorMessage: "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.",
requestParameters: {
  bucketName: REDACTED,
  Host: "s3.amazonaws.com",
  key: "uploads/2b06f950-5d78-4acb-b038-20f0fe80b646/artifacts/9d2e106a-af5a-4752-9e68-8f48f0f5bef2/generic_test_doc.pdf"
},
responseElements: null,
additionalEventData: {
  CipherSuite: "ECDHE-RSA-AES128-GCM-SHA256",
  bytesTransferredIn: 0,
  x-amz-id-2: "uXcDBK3v4NA5LT4Abq0gskfZtncEpI5nIjyGYAwZdJ1VYJ4gxLr5NEU9biEKz500Sw98PprHVJs=",
  bytesTransferredOut: 411
},
<snip>

The environment variables are all correct - I've triple-checked them, and with the fog-aws gem those credentials successfully uploaded an object. Initially I was getting a 400 Bad Request with fog-aws, but then I added the aws_signature_version: 4 to fog_attributes and I instead started getting 403 Forbidden errors with the same "anonymous" credentials showing up in the logs. That's when I tried carrierwave-aws, and it immediately went back to getting the 400 Bad Request response with the logs claiming the signature is wrong as discussed above.

I'm beating my head against the wall here - everything worked fine before we turned on encryption, but AWS KMS seems to have broken our downloads, and unfortunately due to new customers' security requirements, turning off encryption is not an option.... any help or pointers you can lend would be massively appreciated!

sorentwo commented 4 years ago

The aws_attributes directly translate to options in Aws::S3::Object#put, which are explained in the official S3 docs here: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method

For you, the relevant options look to be:

server_side_encryption: "AES256", # accepts AES256, aws:kms and ssekms_key_id: "SSEKMSKeyId",

Anyhow, take a look in the official S3 docs to find the correct combination of options.

jacob-ablowitz commented 4 years ago

I changed the config file attributes to reflect the suggestion you made, it did not work.

I apologize but I neglected to mention that I'm able to upload the file using fog - Object#put works. The only problem is with Object#get.

Additional context: I didn't write the code to do it this way, but the current process works as follows, and it works fine for unencrypted AWS buckets:

React front end client requests a signed URL from api server (works)
Api server uses Fog::storage.new to generate a signed URL and send that signed URL to the react client (works)
React client directly PUTs the file to the S3 bucket (works)
React sends the unsigned URL of the successfully-stored-to-AWS file back to Api Server (works)
Api Server instantiates a new CarrierWave/ActiveRecord object, artifact_item, corresponding to the newly-uploaded file and sets attributes, the last of which is setting remote_file_url with the unsigned AWS URL returned from the front-end
Upon calling artifact_item.save Rails proceeds to create the object in the database and run validations
DownloadValidator fails and returns 400 Bad Request
If I visit S3 storage access logs and/or CloudTrail object logs and search like crazy, sometimes I can find the access request through all the other noise. When I've been able to do so, the log confirms everything is right and proper on the upload side, but the error message says I'm not using signature v4 as previously discussed.

New config/initializers/carrierwave.rb:

  config.storage    = :aws
  config.aws_bucket = ENV.fetch('S3_BUCKET_NAME')
  config.aws_acl    = 'private'
  config.aws_authenticated_url_expiration = S3_URL_EXPIRATION

  config.aws_attributes = -> { {
    server_side_encryption: 'aws:kms',
    sse_kms_key_id:         ENV['S3_KMS_KEY_ID']
  } }

  config.aws_credentials = {
    access_key_id:     ENV.fetch('S3_KEY'),
    secret_access_key: ENV.fetch('S3_SECRET'),
    region:            ENV.fetch('S3_REGION','us-east-1'), # Required
    stub_responses:    Rails.env.test? # Optional, avoid hitting S3 actual during tests
  }

I'm sorry for cluttering up your issues log if this isn't a carrierwave-aws problem, but it seems almost certain to me that somewhere along the line it's reverting to an older signature version, and I can't for the life of me figure out why.

Tests are running inside a docker container, which I've rebuilt a few times. I've tried to make sure I'm up-to-date on the relevant gems - Gemfile.lock extract, filtered to relevant items below. Unfortunately until we get all our automated system end-to-end tests written and integrated, I'm not able to upgrade from Rails 5.1.7 / Ruby 2.4.1 because I can't be certain it won't break things.

    <snip>
    aws-eventstream (1.0.3)
    aws-partitions (1.270.0)
    aws-sdk-core (3.89.1)
      aws-eventstream (~> 1.0, >= 1.0.2)
      aws-partitions (~> 1, >= 1.239.0)
      aws-sigv4 (~> 1.1)
      jmespath (~> 1.0)
    aws-sdk-kms (1.28.0)
      aws-sdk-core (~> 3, >= 3.71.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-s3 (1.60.1)
      aws-sdk-core (~> 3, >= 3.83.0)
      aws-sdk-kms (~> 1)
      aws-sigv4 (~> 1.1)
    aws-sigv4 (1.1.0)
      aws-eventstream (~> 1.0, >= 1.0.2)
    <snip>
    carrierwave (1.2.3)
      activemodel (>= 4.0.0)
      activesupport (>= 4.0.0)
      mime-types (>= 1.16)
    carrierwave-aws (1.4.0)
      aws-sdk-s3 (~> 1.0)
      carrierwave (>= 0.7, < 2.1)

sorentwo commented 4 years ago

Sorry, I wish I had more insight into your issue. I'm not especially familiar with s3 encryption settings—we have encryption enabled for all buckets but nothing is configured on the client side.

I'd try fetching the object directly using Aws::S3::Object#get with various options to see if that works. The options are limited compared to #put, and they don't seem to correspond to the ssemks options.

Side note, above you have: sse_kms_key_id: ENV['S3_KMS_KEY_ID'], but the documented attribute is ssekms_key_id.

jacob-ablowitz commented 4 years ago

Thanks for the typo catch, but unfortunately it didn't fix the problem.

I'm not trying to do client-side encryption, just server-side.

AWS rejects GetObject requests for server-side-encrypted objects unless they are using signature v4.

I just want the built-in get methods to use the correct signature version - at least in my configuration, according to the best S3 logs I can find, the built-in CarrierWave calls end up using signature v2 despite my best efforts to correct that behavior.

I've been trying to figure out where the selection of v2 vs. v4 takes place and look at how that's happening, but I haven't found it yet... even some pointers where to look around would be helpful - if it turns out to actually be a bug in the gem and I can figure out how to fix it, I'm happy to PR my fix...

sorentwo commented 4 years ago

I just want the built-in get methods to use the correct signature version - at least in my configuration, according to the best S3 logs I can find, the built-in CarrierWave calls end up using signature v2 despite my best efforts to correct that behavior.

I've been trying to figure out where the selection of v2 vs. v4 takes place and look at how that's happening, but I haven't found it yet... even some pointers where to look around would be helpful - if it turns out to actually be a bug in the gem and I can figure out how to fix it, I'm happy to PR my fix...

The read function, which is used for get calls, is found in aws_file.rb. That calls read_options, which is found in aws_options.rb.

Hope that helps (and sorry that I can't be of more help).

carrierwaveuploader / carrierwave-aws

uploader fails, returns 400 Bad Request errors #157