search

ostinelli / apnotic

A Ruby APNs HTTP/2 gem able to provide instant feedback.
MIT License
477 stars 94 forks source link

SocketError: Socket was remotely closed #73

Closed datdojp closed 3 years ago

datdojp commented 6 years ago

In the last 2 days, our production server has been raising "SocketError: Socket was remotely closed" whenever it tries to push to APN server. I didn't update production source code for weeks. It suddenly happens on all environments: production, staging, development. I also checked all Apple certificates. They are valid (not expired).

I am not alone. There are 2 people reporting about this issue on Apple forums: https://forums.developer.apple.com/thread/106078

I really appreciate if you know any thing about this issue.

maciejkowalski commented 6 years ago

Yes, it happens from time to time. I switched from ConnectionPool to 1 connection per worker (create new connection, use it to send notifiactions, close it) and it works.

laura-levelup commented 6 years ago

@datdojp - I started the Apple developer forums thread you mentioned. I called into support (1-800-692-7753) and they said that not enough people had reported the issue to them so they do not believe it is an issue on their side

Yes, it happens from time to time.

@maciejkowalski I'm experiencing it consistently now - I have not been able to send a successful single push notification for any of our apps. Are you currently able to send pushes successfully on your end?

ostinelli commented 6 years ago

All info is in https://github.com/ostinelli/apnotic/issues/68.

TL;DR: apple changed their HTTP2 implementation and started enforcing headers ordering, at first only in the development environment. This has already been fixed in net-http2 2 weeks ago. Today, they started enforcing the same thing in production. Just upgrade apnotic & net-http2. Note that you now will need to add a block to your connection pool if you use it, i.e.

APNOTIC_POOL = Apnotic::ConnectionPool.new({
  cert_path: Rails.root.join("config", "certs", "apns_certificate.pem"),
  cert_pass: "mypass"
}, size: 5) do |connection|
  connection.on(:error) { |exception| puts "Exception has been raised: #{exception}" }
end

Check README for info.

laura-levelup commented 6 years ago

Thank you @ostinelli !

maciejkowalski commented 6 years ago

@maciejkowalski I'm experiencing it consistently now - I have not been able to send a successful single push notification for any of our apps. Are you currently able to send pushes successfully on your end?

Yes. 95% of them are received instantly, some of them are later (minutes). Sometimes I receive notification longly (hours) after Sidekiq worker done its job.

datdojp commented 6 years ago

I upgraded apnotic and everything works fine now. So many thanks for all your support.

By the way, I wonder how we prevent this kind of problem in the future. I mean how we know that Apple released their changes so that we can adapt our system accordingly. Do we need to subscribe to a forums, blog, channel or something like that?

benoittgt commented 6 years ago

Thanks a lot for this issue and update on net-http2.

This bug was a real pain for us since two days.

By the way, I wonder how we prevent this kind of problem in the future. I mean how we know that Apple released their changes so that we can adapt our system accordingly. Do we need to subscribe to a forums, blog, channel or something like that?

And I have the same question.

ostinelli commented 6 years ago

By the way, I wonder how we prevent this kind of problem in the future. I mean how we know that Apple released their changes so that we can adapt our system accordingly. Do we need to subscribe to a forums, blog, channel or something like that?

If you find a way let me know. @byronformwalt is also trying to see if anyone knows how to get this kind of information in advance.

FWMatt commented 5 years ago

Hi @ostinelli thank you for the great work on this library.

We have latest apnotic release & net-http2 release, but are still experiencing this issue periodically (lets say 4-5 times per day).

Gemfile.lock

apnotic (1.4.1)
connection_pool (~> 2.0)
net-http2 (>= 0.18, < 2)

Stacktrace

- 6 non-project frames
1
File "/app/vendor/ruby-2.4.4/lib/ruby/2.4.0/openssl/buffering.rb" line 181 in sysread_nonblock
2
File "/app/vendor/ruby-2.4.4/lib/ruby/2.4.0/openssl/buffering.rb" line 181 in read_nonblock
3
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 141 in block in socket_loop
4
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 138 in loop
5
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 138 in socket_loop
6
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 110 in block (2 levels) in ensure_open
SocketError: Socket was remotely closed
- 3 non-project frames
1
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 130 in callback_or_raise
2
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 115 in rescue in block (2 levels) in ensure_open
3
File "/app/vendor/bundle/ruby/2.4.0/gems/net-http2-0.18.0/lib/net-http2/client.rb" line 109 in block (2 levels) in ensure_open

Resending notifications with same payload/config does not cause the issue to occur again.

Any thoughts on how we could provide further diagnosis?

Thank you Matt

benoittgt commented 5 years ago

I don't think you can have more info than this one. See https://github.com/ostinelli/apnotic/issues/41#issue-203624698

FWMatt commented 5 years ago

Thanks for the update - let me know if you'd like me to move my comment to the other issue.

felipefava commented 5 years ago

Hi there! We are using Apnotic in production since one week and it works fine, only that it fails sporadically with SocketError: Socket was remotely closed

I have seen the other issues but we are using the latest versions.

Gemfile.lock:

apnotic (1.5.0)
     connection_pool (~> 2)
     net-http2 (>= 0.18, < 2)
connection_pool (2.2.2)
net-http2 (0.18.2)
    http-2 (~> 0.10.1)

We are also using sync pushes since we have a dedicated project.

Here is the backtrace: 

SocketError: Socket was remotely closed
/usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:130:in `callback_or_raise' 
/usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:115:in `rescue in block (2 levels) in ensure_open'
/usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:109:in `block (2 levels) in ensure_open'

Thanks very much for the help, and awesome work with this library

byronformwalt commented 5 years ago

Hi. APNS will remotely close connections for a variety of reasons (inactivity, server maintenance, server scaling). In my experience, even an inactive connection can remain open for many hours before being remotely closed. As far as I know, Apple does not specify the conditions that result in remotely closing connections. Can you post additional details about your case (e.g. stats on time before experiencing the remote connection is closed)?

On Jun 18, 2019, at 9:00 AM, Felipe Fava notifications@github.com wrote:

Hi there! We are using Apnotic in production since one week and it works fine, only that it fails sporadically with SocketError: Socket was remotely closed

I have seen the other issues but we are using the latest versions.

Gemfile.lock:

apnotic (1.5.0) connection_pool (~> 2) net-http2 (>= 0.18, < 2) connection_pool (2.2.2) net-http2 (0.18.2) http-2 (~> 0.10.1) We are also using sync pushes since we have a dedicated project.

Here is the backtrace:

SocketError: Socket was remotely closed /usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:130:in callback_or_raise' /usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:115:inrescue in block (2 levels) in ensure_open' /usr/local/bundle/gems/net-http2-0.18.2/lib/net-http2/client.rb:109:in `block (2 levels) in ensure_open' Thanks very much for the help, and awesome work with this library

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ostinelli/apnotic/issues/73?email_source=notifications&email_token=AAH2RJDO5YKNKKTNEBG6IZTP3EBDXA5CNFSM4FKUNX42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX7D2WQ#issuecomment-503201114, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH2RJC4FWQVKWJVSI5JBKTP3EBDXANCNFSM4FKUNX4Q.

jdsampayo commented 5 years ago

It appears the issue is on net-http2, using it alone fails sporadically with Socket was remotely closed (SocketError), retrying exactly the same request multiple times until the socket is not closed worked for me. Once the first request is "accepted" all the others never fail on that active connection. It can be the initial negotiation between the servers?

hajimeaiizuka commented 3 years ago

Hi. Is there a possibility that this case will be fixed with net-http2 timeout support? https://github.com/ostinelli/apnotic/commit/e30528bdd7194a183a9bbf235e98a3410f43bf80

benubois commented 3 years ago

This issue is around header ordering, which has been fixed. Any other issues with the same exception name should be re-opened as independent issues. It's also important to use .on(:error) with the connection pool so closed connection can be caught and retried.