Signature mismatch when uploading to DeviceFarm (Chunked Request)

[raymondhoagland]

Issue description

I'm unable to upload a file to Device Farm using a chunked request through excon because I'm getting an error that the computed signature does not match. I'm trying to use a chunked request to avoid having to send the entire request at once - the file being uploaded is large enough that calling File.read directly on it causes an EINVAL error and reading it line by line and sending as one request breaks on the openssl library.

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk

Version of Ruby, OS environment

Ruby 2.3.0, High Sierra

Code snippets / steps to reproduce

I'm currently running the equivalent of the following, the broken pipe handling is needed for me to even see the signature error otherwise I just get a broken pipe error after ~ 2s which I assumed was a result of the mismatch.

require 'excon'

client = ::Aws::DeviceFarm::Client.new

path = File.expand_path('./app-20180707-190539.ipa')

upload_obj = client.create_upload({
    project_arn: project.arn,
    name: File.basename(path),
    type: type
}).upload
puts "Upload url is: #{upload_obj.url}"

fp = File.open(path)
chunker = lambda do
    fp.read(Excon.defaults[:chunk_size]).to_s
end

puts "Beginning upload"
Excon.defaults[:ssl_verify_peer] = false
connection = Excon.new(upload_obj.url, :connect_timeout => 360)
attempts = 0
begin
    resp = connection.request(:write_timeout => 360, :read_timeout => 360, :method => :put, :request_block => chunker)
    puts resp.inspect
rescue Excon::Error::Socket => ees
    if ees.message && ees.message.downcase.include?('broken pipe')
        puts "Retrying because we saw a broken pipe"
        raise ees unless attempts < 5
        attempts += 1
        retry
    end
ensure
    fp.rewind
end
fp.close

[raymondhoagland]

Okay I believe I've made it through that error by adding omit_default_port => true to the excon connection.

Now I'm seeing an error that the Transfer-Encoding header is unimplemented, which seems to rule out a simple chunked request through excon. The reason I had been looking at chunked requests is at one point I saw an error that said that the Content-Length header wasn't set and Transfer-Encoding wasn't set to chunked (I was incorrectly trying to upload the file as multipart form-data). If it's not supported that seems a bit misleading.

I see in the AWS docs for S3 there is a Content-Encoding header which can be set to aws-chunked to provide chunking functionality - is that supported for Device Farm/the aws-sdk-ruby gem? If so, is there an example/way of doing so in Ruby that someone could point out? Or am I going in the wrong direction trying to get a chunked request go work?

[awood45]

I can take another look at this Monday, but I suspect the path forward is to reach out to AWS Support for DeviceFarm. It looks like the SDK is correctly handling things, but you're looking for help with your hand-written HTTP traffic with DeviceFarm.

[srchase]

@raymondhoagland

I'm going to close this issue. If you have evidence that there is an issue with the way the SDK is handling this, let us know and we can re-open.