search

ably / ably-go

Go client library SDK for Ably realtime messaging service
https://ably.com/download
Apache License 2.0
80 stars 32 forks source link

http2 GOAWAY stopping client from reconnecting #605

Open joshforbes opened 1 year ago

joshforbes commented 1 year ago

We're getting this error:

Which is stopping the client and leaving our agent in an offline state. The token endpoint is nothing special running behind an nginx server. Not sure if there is something we should change or if something is going on with the client. Any tips appreciated.

Thanks!

┆Issue is synchronized with this Jira Task by Unito

sacOO7 commented 1 year ago

Hi @joshforbes, thanks for raising the issue! I have some questions before this issue is resolved

  1. ably-go version
  2. go version
  3. Which type of auth mechanism are you using?
  4. Is this only happening with the latest version of ably-go or do previous versions also have the same issue?
  5. If you have a small working snippet that would be able to reproduce this issue, it would be super useful for us to resolve it asap!
joshforbes commented 1 year ago
  1. ably-go 1.2.12
  2. Go 1.20
  3. Token
  4. We upgraded from ably-go 1.1.3 🫣. Things were a lot different and we had years of workarounds to problems.
  5. Sorry, I don't. This one is a bit more common than the other issue I opened. After running for two days on 83 devices we have 3 of them stuck in this state.

We're considering replacing the HTTP client in the Ably client to disable HTTP/2.

joshforbes commented 1 year ago

I appreciate that the Ably client allowed us to so easily replace the HTTP client. We decided to move forward with swapping to an HTTP/1.1 client. I think we'll probably stick with that unless you have some ideas about how to fix the HTTP/2 errors.

sacOO7 commented 1 year ago

Sure, if HTTP/2 is causing errors, you can keep HTTP/1.1 as a default client. We will need to see HTTP2 support for ably-go

sacOO7 commented 4 months ago

Related issue -> https://github.com/olivere/elastic/issues/1443

sync-by-unito[bot] commented 4 months ago

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-4120

sacOO7 commented 4 months ago

We need to add retry mechanism to fallback endpoint when this error occurs. More details for the issue given here https://github.com/connectrpc/connect-go/issues/672 Need to modify canFallback method to check same error in the response, if it's GOAWAY then return true https://github.com/ably/ably-go/blob/dd8236fc301eb078803e83efee7615f3ba133341/ably/rest_client.go#L801-L806