StreamClosedByServerException: Stream closed by server

[Depi1993]

Launch Darkly Configuration:

implementation group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '6.0.4'

With Config:

new LDConfig.Builder() .http(Components.httpConfiguration() .connectTimeout(Duration.ofSeconds(LD_CONNECT_TIMEOUT_SECONDS)) .socketTimeout(Duration.ofSeconds(LD_SOCKET_TIMEOUT_SECONDS))) .events(Components.sendEvents().flushInterval(Duration.ofSeconds(FLUSH_INTERVAL))) .build();

new LDClient(credentials.get(ApiConstants.LD_CLIENT_CREDS_KEY), ldConfig);

After a while of running, started getting - [Multiple Times]

com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188)

 

Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server   |  

[louis-launchdarkly]

Hello @Depi1993,

Would you be able to create a support ticket through https://support.launchdarkly.com ? We likely need to ask you for more information to debug this behavior.

[sourikghosh]

We are facing the same issue with java-sdk

Launch Darkly Configuration: implementation("com.launchdarkly:launchdarkly-java-server-sdk:6.1.0")

ERR log

20:39:22.028 [LaunchDarkly-streaming] INFO com.launchdarkly.sdk.server.LDClient.DataSource - Waiting 852 milliseconds before reconnecting
20:39:22.028 [LaunchDarkly-streaming] WARN com.launchdarkly.sdk.server.LDClient.DataSource - Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
at java.lang.Thread.run(Unknown Source) ~[?:?]
at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:199) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130) ~[reverse-sync.jar:?]
at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272) ~[reverse-sync.jar:?]
com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
20:39:22.027 [LaunchDarkly-streaming] WARN com.launchdarkly.sdk.server.LDClient.DataSource - Encountered EventSource error: {}

[saurabhygk]

We are having the same issue Error in stream connection (will retry): com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server, and we can see these logs multiple times.

[aravindparappil46]

+1 for this. Seeing the same issue. Also seeing the below along with it

com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamIOException: java.net.SocketTimeoutException: timeout
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:132) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571) ~[launchdarkly-java-server-sdk-6.0.4.jar:6.0.4]

Using launchdarkly-java-server-sdk-6.0.4

Functionality does not seem to be affected since flag evaluations are working just fine. But the exception lights up our alerts and had to explicitly ignore it. Would be nice if the SDK itself suppresses/handles that exception

[prashil-g]

We are hitting the same issue for multiple services everyday. I've raised support ticket also We use - compile group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '5.10.7'

com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272)
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130)
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109)
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598)
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58)
    at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571)
    at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188)
    at java.lang.Thread.run(Thread.java:750)

[prashil-g]

I reached out to support and they mentioned to upgrade to latest SDK. but I am seeing issues with even 6.2.1

We are hitting the same issue for multiple services everyday. I've raised support ticket also We use - compile group: 'com.launchdarkly', name: 'launchdarkly-java-server-sdk', version: '5.10.7'

com.launchdarkly.shaded.com.launchdarkly.eventsource.StreamClosedByServerException: Stream closed by server
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.getNextChunk(EventParser.java:272)
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.tryNextEvent(EventParser.java:130)
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventParser.nextEvent(EventParser.java:109)
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.requireEvent(EventSource.java:598)
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource.access$1200(EventSource.java:58)
  at com.launchdarkly.shaded.com.launchdarkly.eventsource.EventSource$IteratorImpl.hasNext(EventSource.java:571)
  at com.launchdarkly.sdk.server.StreamProcessor.lambda$start$1(StreamProcessor.java:188)
  at java.lang.Thread.run(Thread.java:750)

[louis-launchdarkly]

Hello @sourikghosh, @saurabhygk, @aravindparappil46, @prashil-g,

If you don't mind, can you all tell me your server location? I am wondering if is this a geographical-related issue.

Typically, this will happen when there is a deployment on the LaunchDarkly side, which will stop the SDK's connection temporarily, and SDK will retry and re-establish the connection and not impact any flag usage (The SDK will keep serving flags using the data already in memory, and will update once the new connection is up).

[prashil-g]

the service where we use sdk are in multiple regions - eu-west-1, eu-central-1, us-west-2, us-east-2. we see issue in all of them

[JeremyWhiteDev]

I also am having this same issue. LaunchDarkly will be working fine, but occasionally these errors are popping up in the logs. Even when running the application locally from my IDE. I'm using Java SDK 6.2.1 in a Spring Boot application.

[louis-launchdarkly]

I have discussed this with the team, and while this is not a sign of a real issue, Java is not handling the behavior at the correct level and creating a much noisier log, probably causing annoyance with people who try to directly have the logs triggering alerts. We will look into a way to capture this information in a less noisy way (as the SDK will correctly reconnect when it senses a disconnect).

[prashil-g]

@louis-launchdarkly This is causing operational issue atleast for us. we see that after this error happens, SDK is not reconnecting on its own and new variations are not being pulled

LaunchDarkly data source outage - updates have been unavailable for at least 1 minute with the following errors: NETWORK_ERROR (1 time)

[louis-launchdarkly]

Hello @prashil-g, can you create a ticket via through https://support.launchdarkly.com/ ? Your error looks different from the others - in particular, it doesn't seem like the Stream closed by server issue. We need more information like your SDK configuration and your network topology to debug.

[mitchellos]

We're seeing this too using version 6.2.1. Seems like things continue to work but yes, logging is triggering alerts. Would be great if this could be resolved!

Looks like the exception is coming from okhttp here, but called from the LD client here?

[github-actions[bot]]

This issue is stale because it has been open for 90 days without activity. Remove the stale label or comment, or this will be closed in 14 days.