search

launchdarkly / go-server-sdk

LaunchDarkly Server-side SDK for Go
Other
41 stars 18 forks source link

Streaming does not work on aws lambda #37

Closed helmus closed 2 years ago

helmus commented 4 years ago

Is this a support request? No

Describe the bug Streaming does not work on AWS lambda

To reproduce use streaming on lambda If you disable streaming using config.Stream = false everything works as expected

Expected behavior Streaming should work

Logs [LaunchDarkly] 2020/08/06 00:35:33 WARN: Unexpected error while sending events: Post https://events.intuit.launchdarkly.com/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [LaunchDarkly] 2020/08/06 00:35:03 WARN: Will retry posting events after 1 second [LaunchDarkly] 2020/08/06 00:35:03 WARN: Unexpected error while sending events: Post https://events.intuit.launchdarkly.com/bulk: EOF [LaunchDarkly] 2020/08/06 00:11:36 ERROR: Network error on streaming connection: Read timeout on stream

SDK version 2a44fb3618958f849648319cc1a500ed0b52d6b9

Language version, developer tools go 1.13

OS/platform AWS lambda

Additional context If you disable streaming using config.Stream = false everything works as expected

eli-darkly commented 4 years ago

It looks like you're one of our customers who has a dedicated LaunchDarkly instance, and it also looks like this is a network connectivity issue. That's outside the scope of what the SDK team can troubleshoot; I think the best way to go would be through the usual support route, either support.launchdarkly.com or any other contacts that your company has. This is highly unlikely to be related to the SDK code.

eli-darkly commented 4 years ago

Assuming it is a network connectivity issue, I'm not sure why it would not also cause problems in polling mode (which is what I think you're saying - did it actually succeed in getting feature flags when you turned off config.Stream, or do you just mean that you didn't see these specific errors?). But it's clearly not just for the streaming connection, since you can see that requests to the events service are also failing. In any case, I think this really is a support request.

helmus commented 4 years ago

Retrieving feature flags succeeded without issues when streaming is disabled. When streaming is enabled, the sdk client also initialized successfully, but then fails to retrieve any flags. These are the start-up messages when streaming is enabled:

[LaunchDarkly] 2020/08/05 23:45:36 INFO: Starting LaunchDarkly streaming connection
[LaunchDarkly] 2020/08/05 23:37:38 INFO: LaunchDarkly streaming is active
[LaunchDarkly] 2020/08/05 23:38:24 ERROR: Network error on streaming connection: Get https://stream.intuit.launchdarkly.com/all: dial tcp 52.24.136.58:443: i/o timeout
eli-darkly commented 4 years ago

Actually, I can think of a reason why the polling requests would not be affected— I presume there are restrictions on connecting to the Intuit hosts from external systems, and it's likely that those restrictions are somewhat different from one service to another based on expectations about what is going to be connecting to them (the polling service would have to accept requests from browsers and mobile clients). But again I strongly recommend that you go through support, because the SDK team has no knowledge of Intuit deployment issues beyond the very general things I just mentioned. I don't know whether any of your teams have previously used our SDKs from AWS in this way and if so, whether they had to do anything special to make it work. It is common to use LaunchDarkly services from AWS-hosted applications, but you're not using the standard LaunchDarkly services, you have a custom installation.

eli-darkly commented 4 years ago

It's also possible that this is just due to the Lambda execution context being suspended once your function is not actively doing things - that may be a wrong guess, since I don't know how the timestamps of those log messages compare to the lifespan of your Lambda (the timestamps also appear to be out of order), but long-lived network connections are problematic in Lambda for that reason. That wouldn't account for the event posts failing though.

eli-darkly commented 2 years ago

Closing this because there were no further updates from the user here - it would be one for the support team, due to the special circumstances of this being a dedicated instance.