Excon::Errors::Forbidden with files containing Umlauts

[tvdeyen]

Uploading files to S3 fails if the file has an umlaut in its name.

Excon::Errors::Forbidden - Expected(200) <=> Actual(403 Forbidden)
  response => #<Excon::Response:0x007f90aecc91b0 @data={:body=>"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 70 6e 67 0a 54 68 75 2c 20 30 36 20 46 65 62 20 32 30 31 34 20 31 32 3a 30 36 3a 30 30 20 2b 30 30 30 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6a 73 6f 6e 3a 7b 22 6e 61 6d 65 22 3a 22 70 61 73 73 66 6f 74 6f 2d 6d 69 74 2d 6c c3 83 c2 a4 63 68 65 6c 6e 2e 70 6e 67 22 2c 22 6d 6f 64 65 6c 5f 63 6c 61 73 73 22 3a 22 50 69 63 74 75 72 65 22 2c 22 6d 6f 64 65 6c 5f 61 74 74 61 63 68 6d 65 6e 74 22 3a 22 69 6d 61 67 65 22 2c 22 61 6e 61 6c 79 73 65 72 5f 63 61 63 68 65 22 3a 7b 22 69 6d 61 67 65 5f 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 66 6f 72 6d 61 74 22 3a 22 70 6e 67 22 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 7d 0a 2f 66 61 76 69 63 73 2d 74 65 73 74 69 6e 67 2f 32 30 31 34 2f 30 32 2f 30 36 2f 31 33 2f 30 35 2f 35 39 2f 39 38 39 2f 70 61 73 73 66 6f 74 6f 5f 6d 69 74 5f 6c 5f 63 68 65 6c 6e 2e 70 6e 67</StringToSignBytes><RequestId>BF4085C899FB04BC</RequestId><HostId>BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f</HostId><SignatureProvided>wmJyDEg8Djr9E/Jm1huYjpZmK8A=</SignatureProvided><StringToSign>PUT\n\nimage/png\nThu, 06 Feb 2014 12:06:00 +0000\nx-amz-acl:public-read\nx-amz-meta-json:{\"name\":\"passfoto-mit-l\xC3\x83\xC2\xA4cheln.png\",\"model_class\":\"Picture\",\"model_attachment\":\"image\",\"analyser_cache\":{\"image_properties\":{\"format\":\"png\",\"width\":92,\"height\":128},\"width\":92,\"height\":128}}\n/favics-testing/2014/02/06/13/05/59/989/passfoto_mit_l_cheln.png</StringToSign><AWSAccessKeyId>AKIAJWH23ATSHYAAVEKA</AWSAccessKeyId></Error>", :headers=>{"x-amz-request-id"=>"BF4085C899FB04BC", "x-amz-id-2"=>"BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"", "Date"=>"Thu, 06 Feb 2014 12:06:01 GMT", "Connection"=>"close", "Server"=>"AmazonS3"}, :status=>403, :remote_ip=>"178.236.4.121"}, @body="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 70 6e 67 0a 54 68 75 2c 20 30 36 20 46 65 62 20 32 30 31 34 20 31 32 3a 30 36 3a 30 30 20 2b 30 30 30 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6a 73 6f 6e 3a 7b 22 6e 61 6d 65 22 3a 22 70 61 73 73 66 6f 74 6f 2d 6d 69 74 2d 6c c3 83 c2 a4 63 68 65 6c 6e 2e 70 6e 67 22 2c 22 6d 6f 64 65 6c 5f 63 6c 61 73 73 22 3a 22 50 69 63 74 75 72 65 22 2c 22 6d 6f 64 65 6c 5f 61 74 74 61 63 68 6d 65 6e 74 22 3a 22 69 6d 61 67 65 22 2c 22 61 6e 61 6c 79 73 65 72 5f 63 61 63 68 65 22 3a 7b 22 69 6d 61 67 65 5f 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 66 6f 72 6d 61 74 22 3a 22 70 6e 67 22 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 7d 0a 2f 66 61 76 69 63 73 2d 74 65 73 74 69 6e 67 2f 32 30 31 34 2f 30 32 2f 30 36 2f 31 33 2f 30 35 2f 35 39 2f 39 38 39 2f 70 61 73 73 66 6f 74 6f 5f 6d 69 74 5f 6c 5f 63 68 65 6c 6e 2e 70 6e 67</StringToSignBytes><RequestId>BF4085C899FB04BC</RequestId><HostId>BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f</HostId><SignatureProvided>wmJyDEg8Djr9E/Jm1huYjpZmK8A=</SignatureProvided><StringToSign>PUT\n\nimage/png\nThu, 06 Feb 2014 12:06:00 +0000\nx-amz-acl:public-read\nx-amz-meta-json:{\"name\":\"passfoto-mit-l\xC3\x83\xC2\xA4cheln.png\",\"model_class\":\"Picture\",\"model_attachment\":\"image\",\"analyser_cache\":{\"image_properties\":{\"format\":\"png\",\"width\":92,\"height\":128},\"width\":92,\"height\":128}}\n/favics-testing/2014/02/06/13/05/59/989/passfoto_mit_l_cheln.png</StringToSign><AWSAccessKeyId>AKIAJWH23ATSHYAAVEKA</AWSAccessKeyId></Error>", @headers={"x-amz-request-id"=>"BF4085C899FB04BC", "x-amz-id-2"=>"BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"", "Date"=>"Thu, 06 Feb 2014 12:06:01 GMT", "Connection"=>"close", "Server"=>"AmazonS3"}, @status=403, @remote_ip="178.236.4.121">:

uri encode the filename before storing works:

dragonfly_accessor :image do
  after_assign { |a| a.name = URI.escape(a.name) }
end

[felixbuenemann]

For now I've monkey-patched dragonfly-s3_data_store to escape utf8 in json, which does the trick:

module Dragonfly
  class S3DataStore
    def meta_to_headers(meta)
      {'x-amz-meta-json' => JSON.generate(meta, ascii_only: true)}
    end
  end
end

The default implementation uses MultiJson.encode(meta), which does not escape utf8.

I've opened an issue for fog at fog/fog#2942.

[janraasch]

Hi guys, just chiming in, because I ran into this problem as well. IMHO this really needs to be fixed in fog (see https://github.com/fog/fog/issues/2942), but just for reference I want to post our solution here, since JSON.generate(meta, ascii_only: true) did not work for us.

We decided to monkey patch this in our rails v3 app (running on ruby v2.1.2) with a module

module Dragonfly
  module S3DataStoreUmlauts
    def meta_to_headers(meta)
      { 'x-amz-meta-json' => ActiveSupport::JSON.encode(meta) }
    end
  end
end

which we then use to extend our datastore like so

Dragonfly.app.datastore.extend Dragonfly::S3DataStoreUmlauts

PS: This won't work with rails v4, see https://github.com/rails/rails/commit/8f8397e0a4ea2bbc27d4bba60088286217314807 and https://github.com/rails/rails/issues/3727

[felixbuenemann]

Due to another signing error, when the filename contains multiple spaces I'm now using this:

module Dragonfly
  class S3DataStore
    def meta_to_headers(meta)
      {'x-amz-meta-json' => JSON.generate(meta, ascii_only: true).gsub(' ', '\u0020')}
    end
  end
end

[felixbuenemann]

@markevans Maybe dragonfly should simply base64 encode the json? There are so many cases where the generated headers can break either RFC2616 Section 4.2 or S3's buggy interpretation of those rules.

[felixbuenemann]

Oops, deleted last comment which should've landed on a fog/fog-aws#160 ticket.