Event Source receives heartbeat event but still gives error when timeout: Error: No activity within N milliseconds. 96 characters received. Reconnecting.

[AnhTuan-AiT]

I am working on a web application using react and spring boot. To add live notification feature, I choosed server-sent-events with your library on the client. I set heartbeatTimeout to 120s and periodically send a heartbeat event every 40s to keep the connection open.

It works OK when I test it locally, but when I deploy the app it doesn't anymore. The connection is still open normally, the client still receives the heartbeat events in full and at the right time, but usually every 3 heartbeat events, the client gives an error: Error: No activity within N milliseconds. 96 characters received. Reconnecting.

I think the biggest difference between local environment and my deployment environment is that in local environment, client connects directly to backend, and in deployment environment, I use Nginx between them. But I still can't figure out which part of the deployment pattern is the reason of error.

Here is the code I use: React

React.useEffect(() => {
    let eventSource;
    let reconnectFrequencySeconds = 1;

    // Putting these functions in extra variables is just for the sake of readability
    const wait = function () {
      return reconnectFrequencySeconds * 1000;
    };

    const tryToSetup = function () {
      setupEventSource();
      reconnectFrequencySeconds *= 2;

      if (reconnectFrequencySeconds >= 64) {
        reconnectFrequencySeconds = 64;
      }
    };

    // Reconnect on every error
    const reconnect = function () {
      setTimeout(tryToSetup, wait());
    };

    function setupEventSource() {
      fetchNotification();

      eventSource = new EventSourcePolyfill(
        `${API_URL}/notification/subscription`,
        {
          headers: {
            "X-Auth-Token": store.getState().auth.token,
          },
          heartbeatTimeout: 120000,
        }
      );

      eventSource.onopen = (event) => {
        console.info("SSE opened");
        reconnectFrequencySeconds = 1;
      };

      // This event only to keep sse connection alive
      eventSource.addEventListener(SSE_EVENTS.HEARTBEAT, (e) => {
        console.log(new Date(), e);
      });

      eventSource.addEventListener(SSE_EVENTS.NEW_NOTIFICATION, (e) =>
        handleNewNotification(e)
      );

      eventSource.onerror = (event) => {
        // When server SseEmitters timeout, it cause error
        console.error(
          `EventSource connection state: ${
            eventSource.readyState
          }, error occurred: ${JSON.stringify(event)}`
        );

        if (event.target.readyState === EventSource.CLOSED) {
          console.log(
            `SSE closed (event readyState = ${event.target.readyState})`
          );
        } else if (event.target.readyState === EventSource.CONNECTING) {
          console.log(
            `SSE reconnecting (event readyState = ${event.target.readyState})`
          );
        }

        eventSource.close();
        reconnect();
      };
    }

    setupEventSource();

    return () => {
      eventSource.close();
      console.info("SSE closed");
    };
  }, []);

Spring

    /**
     * @param toUser
     * @return
     */
    @GetMapping("/subscription")
    public ResponseEntity<SseEmitter> events(
        @CurrentSecurityContext(expression = "authentication.name") String toUser
    ) {
        log.info(toUser + " subscribes at " + getCurrentDateTime());

        SseEmitter subscription;

        if (subscriptions.containsKey(toUser)) {
            subscription = subscriptions.get(toUser);
        } else {
            subscription = new SseEmitter(Long.MAX_VALUE);
            Runnable callback = () -> subscriptions.remove(toUser);

            subscription.onTimeout(callback); // OK
            subscription.onCompletion(callback); // OK
            subscription.onError((exception) -> { // Must consider carefully, but currently OK
                subscriptions.remove(toUser);
                log.info("onError fired with exception: " + exception);
            });

            subscriptions.put(toUser, subscription);
        }

        HttpHeaders responseHeaders = new HttpHeaders();
        responseHeaders.set("X-Accel-Buffering", "no");
        responseHeaders.set("Cache-Control", "no-cache");

        return ResponseEntity.ok().headers(responseHeaders).body(subscription);
    }

    /**
     * To keep connection alive
     */
    @Async
    @Scheduled(fixedRate = 40000)
    public void sendHeartbeatSignal() {
        subscriptions.forEach((toUser, subscription) -> {
            try {
                subscription.send(SseEmitter
                                      .event()
                                      .name(SSE_EVENT_HEARTBEAT)
                                      .comment(":\n\nkeep alive"));
//                log.info("SENT HEARBEAT SIGNAL AT: " + getCurrentDateTime());
            } catch (Exception e) {
                // Currently, nothing need be done here
            }
        });
    }

Nginx

events{
}
http {

server {
    client_max_body_size 200M;
    proxy_send_timeout 12000s;
    proxy_read_timeout 12000s;
    fastcgi_send_timeout 12000s;
    fastcgi_read_timeout 12000s;
    location = /api/notification/subscription {
        proxy_pass http://baseweb:8080;
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
        proxy_buffering off;
        proxy_buffer_size 0;
        proxy_cache off;
        proxy_connect_timeout 600s;
        fastcgi_param NO_BUFFERING "";
        fastcgi_buffering off;
    }
}
}

I really need support now

[Yaffle]

The connection is still open normally, the client still receives the heartbeat events in full and at the right time

Sounds, like this is a bug in the library, I would try to debug the js code, please put a break point on the place that throws the exception and looks what is going on.

[AnhTuan-AiT]

Please give me your mail, I will give u my app address and account so u can debug directly

[AnhTuan-AiT]

Please check your mail. I sent the infomation to u

[Yaffle]

@AnhTuan-AiT , nothing

[AnhTuan-AiT]

@Yaffle I don't know the reason but could u send me an email to: anhtuan0126104@gmail.com and I will reply to it

[Yaffle]

@AnhTuan-AiT , it works for me there is an unclear delay before the open event when I am using Firefox

[AnhTuan-AiT]

After trying to find the cause of the error, I obtained the following symptoms.

Before talking about the phenomenon, I will describe how my application works. When the user accesses the application, I will check the token in localStorage, if exists (regardless of whether the token is valid or not) then the application's state will be set as logged in and the component containing the Event Source object will be rendered and initiate a connection to the server with an expired token placed in the headers. And of course it can't connect, then the onerror callback function will be called and the reconnect function will be called. And here is the next phenomenon:

1. When I press the login button, the application goes to the login screen and of course the component containing the Event Source object is unmounted, but when I look at the console, what I see is the reconnect function keeps executing.

2. I have a theory and to test the theory, I added a flag value as follows:

   headers: {
         "X-Auth-Token": store.getState().auth.token,
         Count: count++,
       },

   Here, count will be incremented by 1 every time the reconnect function is called. And after I have successfully logged in, got a valid new token, successfully connected to the server, usually every 3 heartbeat events, I get the error: No activity within N milliseconds. 96 characters received. Reconnecting. Thanks to the count variable, I know that the error was thrown from the Event Source object that failed in the previous connection, when I wasn't logged in.

3. When I remove the reconnect. At the time I access the application, of course the Event Source will not be able to connect and the onerror callback function is called, but this time there is no reconnect function so only an Event Source object is created. After logging in, I see the Event Source successfully connected to the server and received the heartbeat events fully and in the correct cycle, no more errors.

I don't know why the reconnect function can continue to run when the component is unmounted and why the previous Event Source objects continue to exist

[AnhTuan-AiT]

@AnhTuan-AiT , it works for me there is an unclear delay before the open event when I am using Firefox

On chrome I also see this phenomenon, maybe this is a backend issue but I don't think it's a big deal. Maybe I'll try to determine the cause and fix this later

[Yaffle]

@AnhTuan-AiT looks like you need to disconnect using the "close" method when you are unmounting the component. I think, It will not be closed automatically because you have listeners attached which print to the console.

[AnhTuan-AiT]

In useEffect, I return a clean up function:

return () => {
      eventSource.close();
      console.info("SSE closed");
    };

Is it not working?

[AnhTuan-AiT]

Or do you mean I have to collect all created EventSource objects and call close() method on all of them

[Yaffle]

no, only on the one you want to close, don't know if useEffect works

[AnhTuan-AiT]

I use the reconnect func just because I want to control the speed of the connection again, avoiding the server being overloaded due to repeated reconnections with short cycles. When I don't use it, as commented above, everything works OK, no errors occur. Because I know there are some cases where the server-sent-event won't try to reconnect again, for example error 500 or 502.

Do you think I really need this reconnect function?

[Yaffle]

looks like you need it

[AnhTuan-AiT]

I will reevaluate the need for this function and possibly remove it. I also want to ask if I use addEventListener for each event instead of using onmessage and handle all events in this function

Does it could be the cause of the error when using the reconnect func?

[Yaffle]

@AnhTuan-AiT , no, usage of onmessage in place of addEventListener should not make any difference