search

jruby / jruby

JRuby, an implementation of Ruby on the JVM
https://www.jruby.org
Other
3.8k stars 925 forks source link

Consistent "Connection Reset" for AWS S3 Client put_object #7795

Closed rcrews closed 2 weeks ago

rcrews commented 1 year ago

Using JRuby 9.4.2.0 and Apple Silicon with the AWS Ruby SDK for S3, uploading an object of more than about 500 KB fails with the message "Broken Pipe" or "Connection Reset" (Seahorse::Client::NetworkingError). I've attached a program that consistently demonstrates this for me. The same program runs without errors on MRI 3.2.2.

The backtrace shows the problem comes from stdlib/net/http.rb:1643:in `transport_request'. I've been able to verify the actual error from http.rb is IOError.

This was not a problem when I was running 9.3. on an Intel mac. (I started using 9.4. when I got an ARM mac.) On my ARM mac, I've tried many different versions of Java (8, 11, 17, 19, 20) and different distributions (OpenJDK, Temurin). All ultimately yield the same result. I've also tried different versions of JRuby, but not as rigorously. I'm not sure this is a JRuby 9.4 issue or a Java-on-Apple-Silicon issue. Key take-away: the problem is always for the AWS S3 Client put_object method. I'm not having trouble with various other operations, such as get_object. Also, this same script works with MRI on the same computer, which is why I don't think it is specifically an AWS Ruby SDK bug.

Environment Information

% jruby -v
jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +jit [arm64-darwin]

% uname -a
Darwin Roberts-MacBook-Pro.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:58 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6020 arm64

To test, you'll need an AWS s3 bucket and credentials giving you permission to write to it.

Demonstration Script

#!/usr/bin/env ruby -w

require 'aws-sdk-s3'
require 'digest'
require 'json'

bucket = 'aws-s3-bucket-name' # Replace with your bucket name
key = 'hobbit.txt'
body = <<~CONTENT
  In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet
  hole, filled with the ends of worms and an oozy smell, nor yet a dry,
  bare, sandy hole with nothing in it to sit down on or to eat: it was a
  hobbit-hole, and that means comfort.
CONTENT

client = Aws::S3::Client.new()

client.put_object(bucket:, key:, body:)

opening = {}
(1..1000).each do |i|
  opening[i.to_s] = client.get_object(bucket:, key:).body.read
end
my_json = opening.to_json
p my_json

begin
  json_key = "#{File.basename(key, '.*')}.json"
  client.put_object(body: my_json,
                    bucket:,
                    content_md5: Digest::MD5.base64digest(my_json),
                    content_type: 'application/json',
                    key: json_key)
  puts "Upload: #{URI.join("s3://#{bucket}", json_key)}"
rescue StandardError => e
  puts "#{e.class}: #{e.message}", e.backtrace
end
headius commented 1 year ago

Can you give more information about the error? We will try to reproduce this week to fix for 9.4.3 but I might be able to wager a guess given the full error info.

rcrews commented 1 year ago

Here's the backtrace, plus my editorializing:

Roberts-MacBook-Pro 21:31 ~/Sandbox/navigation % bundle exec exe/aws_s3_upload.rb                                                                                                                                                    git:(main|+5^1 
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/ostruct.rb:467: warning: OpenStruct#public_send accesses caller method's state and should not be aliased
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/ostruct.rb:467: warning: OpenStruct#method accesses caller method's state and should not be aliased
... irrelevant "my_json" data printed here ...
Seahorse::Client::NetworkingError: Connection reset
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1643:in `transport_request'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1579:in `request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/connection_pool.rb:348:in `request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/handler.rb:133:in `block in session'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/handler.rb:128:in `session'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/handler.rb:50:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/content_length.rb:24:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/request_callback.rb:87:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/s3_signer.rb:73:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/xml/error_handler.rb:10:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/sign.rb:49:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/s3_signer.rb:48:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:394:in `retry_request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:382:in `retry_if_possible'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:394:in `retry_request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:382:in `retry_if_possible'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:394:in `retry_request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:382:in `retry_if_possible'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/md5s.rb:31:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:23:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:21:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/rest/handler.rb:10:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/endpoints.rb:41:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/request.rb:72:in `send_request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-s3-1.122.0/lib/aws-sdk-s3/client.rb:12650:in `put_object'
/Users/rcrews/Sandbox/navigation/exe/aws_s3_upload.rb:29:in `<main>'

I think the critical info is this line:

/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1643:in `transport_request'

When I look at that method in http.rb, I see the "raise" comes from this block:

rescue Net::ReadTimeout, IOError, EOFError,
       Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE, Errno::ETIMEDOUT,
       # avoid a dependency on OpenSSL
       defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError,
       Timeout::Error => exception
  if count < max_retries && IDEMPOTENT_METHODS_.include?(req.method)
    count += 1
    @socket.close if @socket
    debug "Conn close because of error #{exception}, and retry"
    retry
  end
  debug "Conn close because of error #{exception}"
  @socket.close if @socket
  raise # <-- Line 1643
end

For this report, I added a debugging line just above the "raise":

rescue Net::ReadTimeout, IOError, EOFError,
       Errno::ECONNRESET, Errno::ECONNABORTED, Errno::EPIPE, Errno::ETIMEDOUT,
       # avoid a dependency on OpenSSL
       defined?(OpenSSL::SSL) ? OpenSSL::SSL::SSLError : IOError,
       Timeout::Error => exception
  if count < max_retries && IDEMPOTENT_METHODS_.include?(req.method)
    count += 1
    @socket.close if @socket
    debug "Conn close because of error #{exception}, and retry"
    retry
  end
  debug "Conn close because of error #{exception}"
  @socket.close if @socket
  puts "#{exception.class}: #{exception.message}", exception.backtrace # rpc
  raise
end

and got this additional info, which seems to give the side-eye to org/jruby/ext/openssl/SSLSocket.java:

IOError: Connection reset
org/jruby/ext/openssl/SSLSocket.java:877:in `sysread_nonblock'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/openssl/buffering.rb:205:in `read_nonblock'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/protocol.rb:212:in `rbuf_fill'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/protocol.rb:193:in `readuntil'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/protocol.rb:203:in `readline'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http/response.rb:42:in `read_status_line'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http/response.rb:31:in `read_new'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1615:in `block in transport_request'
org/jruby/RubyKernel.java:1292:in `catch'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1606:in `transport_request'
/Users/rcrews/.rvm/rubies/jruby-9.4.2.0/lib/ruby/stdlib/net/http.rb:1579:in `request'
/Users/rcrews/.rvm/gems/jruby-9.4.2.0@navigation/gems/aws-sdk-core-3.172.0/lib/seahorse/client/net_http/connection_pool.rb:348:in `request'
... etc. same as above
rcrews commented 1 year ago

If you're wondering... Seahorse::Client::NetworkingError is not my code. "Seahorse" is the project name for the scriptable AWS S3 client, documented at Seahorse::Client. For the purposes of this report, you can just lump that line in with the other aws-sdk-* lines...

headius commented 4 months ago

Never got around to this and will have to punt again. We could use some help here, but otherwise I'll get to it when I can!

rajaravivarma-r commented 3 months ago

I'm facing this issues as well, but it my case it is always IOError: Broken pipe. Attaching the backtrace for the error,

/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/stdlib/net/http.rb:1602:in `block in transport_request'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/stdlib/net/http.rb:1600:in `catch'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/stdlib/net/http.rb:1600:in `transport_request'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/stdlib/net/http.rb:1573:in `request'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/gems/shared/gems/rest-client-2.1.0/lib/restclient/request.rb:471:in `net_http_do_request'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/gems/shared/gems/rest-client-2.1.0/lib/restclient/request.rb:733:in `block in transmit'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/stdlib/net/http.rb:985:in `start'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/gems/shared/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/gems/shared/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
/Users/rajaravivarma/.asdf/installs/ruby/jruby-9.4.6.0/lib/ruby/gems/shared/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'

The same will work as expected once in a while.

My environment:

jruby -v
jruby 9.4.6.0 (3.1.4) 2024-02-20 576fab2c51 OpenJDK 64-Bit Server VM 22+35-2369 on 22+35-2369 [arm64-darwin]
uname -a
Darwin admins-MacBook-Pro.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:13:18 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6030 arm64
headius commented 2 weeks ago

@rcrews A whole year later and I finally got around to attempting this... but I cannot reproduce with latest aws-s3 gem and latest JRuby.

I set up a bucket as described and have run your example script five times. Each time it successfully prints out a ball of json and quietly exits without raising an error.

I am on MacOS AArch64 on JDK 8 with JRuby 9.4 HEAD (which will soon be 9.4.9).

If you are able to reproduce with latest everything, please re-open and we can help figure out what's wrong. For now I think it's reasonable to close this and hope that fixes in JRuby or the net-* libraries fixed this along the way.

headius commented 2 weeks ago

@rajaravivarma-r Your error seems to be different. Could you open a new issue and provide a way for us to reproduce your error?

rajaravivarma-r commented 2 weeks ago

@headius I think most of these errors were arising from SSL Error. When I checked it with my team, they said that cloudfare was terminating those connections based on my IP. It worked fine when I uploaded files from a (Linux) server which had a different public IP address.

I'll create a new ticket if this theory is wrong.

Thank you.

headius commented 2 weeks ago

@rajaravivarma-r Thank you!