Solving the realtime issues

[maxfornacon]

I would like to address the existing issues with realtime (e.g. #705, #579), particularly the problem where streams suddenly stop receiving data. I'm willing to spend some time investigating the source code and will report back if I find anything noteworthy. Since I'm not yet familiar with the repository, I would like to keep this issue as an open discussion. This will allow us to collaborate, so I can seek help if I encounter difficulties or if you believe my research is heading in the wrong direction.

If you have any additional information or thoughts on why this problem might exist, please leave a comment. Perhaps we can finally find a solution to these issues together as a community.

I'm using this repo for my local testing: https://github.com/maxfornacon/supabase_test

I'm using latest supabase_flutter (2.6.0) and Flutter 3.24.0. I'm running this test project on macOS as a desktop application.

The main problems I'm having with realtime in my projects for some time now is: When the application is running for a longer period of time (e.g. desktop applications that are opened (might be in background) the whole day and don't get killed by the system) streams stop receiving new data after some time. There are no errors or exceptions thrown and the only solution I've found is to encourage the users to restart the application from time to time.

The current state of my research is that there might be a problem related to the accessToken refresh. I've set the access token expiry time in supabase dashboard to a small value (30 seconds) and added a breakpoint in this method (realtime_client.dart):

  /// Sets the JWT access token used for channel subscription authorization and Realtime RLS.
  ///
  /// `token` A JWT strings.
  void setAuth(String? token) {
    accessToken = token;

    for (final channel in channels) {
      if (token != null) {
        channel.updateJoinPayload({'user_token': token});
      }
      if (channel.joinedOnce && channel.isJoined) {
        channel.push(ChannelEvents.accessToken, {'access_token': token});
      }
    }
  }

This gets called every time the accessToken is refreshed and should update existing channels with the new token (that's my understanding).

I watched the channels List which contains 3 entries in the beginning for my test project. After some token refreshes this list contains < 3 entries and eventually 0 in the end. This is also when I noticed that the streams don't receive new data anymore.

I don't know yet why the channels get removed from this list but It feels not intended.

That's my current status. I'm curious about your thoughts on this.

[maxfornacon]

Today I checked if the _onClose()-method of the RealtimeChannel gets called and therefore the channel gets removed from the channels List. It does by receiving an event phx_close. I'm not sure what this means. Maybe you can help me out with that. Does it mean the channel gets closed by the server?

This happens after a couple minutes ( with token expiry time of 30 sec). When I set the token expiry time back to 1 hour this didn't happen in that short amount of time.

[Vinzent03]

I wanted to take a look at the realtime issues in the next weeks as well. That event is not listed here, but I found no usage of the event on the client as well, so it's probably sent from the server, yes. Was the application in background or foreground? An interesting event to keep an eye on are sent and received heardbeats. Are you focusing on the raw realtime client or the stream method as well? I would recommend using the realtime client itself to see where the issue exactly lies.

[maxfornacon]

Was the application in background or foreground?

The application is open in "Full Screen" on macOS but the focus was on my IDE. So the output of the AppLifecycleListener was AppLifecycleState.hidden and immediately AppLifecycleState.inactive.

Are you focusing on the raw realtime client or the stream method as well?

Both, but I'm going to switch to testing it with just one from now on. This is how I create the channel:

 Supabase.instance.client
        .channel('notifications')
        .onPostgresChanges(
          event: PostgresChangeEvent.all,
          schema: 'testschema',
          table: 'notifications',
          callback: (payload) {
            print('Change received: ${payload.toString()}');
            if (payload.errors != null && payload.errors.isNotEmpty) {
              payload.errors.forEach((element) {
                print('Error: ${element.message}');
              });
            }
          },
        )
        .subscribe(
            (status, [error]) {
          if (status == RealtimeSubscribeStatus.closed) {
            print('!!!!!! CLOSED !!!!!!!');
          }
        }
    );

[maxfornacon]

I actually couldn't reproduce this problem when I only had one realtime channel.

When I switched back to accessing two (using the stream method), then the problem occurred again.

Maybe #231 isn't working correctly?

EDIT: the channels List being empty after some time also happens when only using only 1 stream. (using the stream method in a Flutter StreamBuilder)

[denghejun]

same issue here, realtime issue: Bad state: Cannot add event after closing flutter: #0 _StreamController.add (dart:async/stream_controller.dart:605) https://github.com/supabase/supabase-flutter/issues/1 _CompleterSink.add (package:web_socket_channel/src/sink_completer.dart:90) https://github.com/supabase/supabase-flutter/issues/2 RealtimeClient.push.callback. (package:realtime_client/src/realtime_client.dart:293) https://github.com/supabase/supabase-flutter/issues/3 new RealtimeClient. (package:realtime_client/src/realtime_client.dart:137) https://github.com/supabase/supabase-flutter/pull/4 RealtimeClient.push.callback (package:realtime_client/src/realtime_client.dart:293) https://github.com/supabase/supabase-flutter/issues/5 RealtimeClient.push (package:realtime_client/src/realtime_client.dart:310) https://github.com/supabase/supabase-flutter/pull/6 RealtimeClient.sendHeartbeat (package:realtime_client/src/realtime_client.dart:482) https://github.com/supabase/supabase-flutter/issues/7 RealtimeClient._onConnOpen. (package:realtime_client/src/realtime_client.dart:400) https://github.com/supabase/supabase-flutter/issues/8 _rootRunUnary (dart:async/zone.dart:1407) https://github.com/supabase/supabase-flutter/issues/9 _CustomZone.runUnary (dart:async/zone.dart:1308) https://github.com/supabase/supabase-flutter/issues/10 _CustomZone.runUnaryGuarded (dart:async/zone.dart:1217) https://github.com/supabase/supabase-flutter/issues/11 _CustomZone.bindUnaryCallbackGuarded. (dart:async/zone.dar<…> flutter: Bad state: Cannot add event after closing flutter: #0 _StreamController.add (dart:async/stream_controller.dart:605) https://github.com/supabase/supabase-flutter/issues/1 _CompleterSink.add (package:web_socket_channel/src/sink_completer.dart:90) https://github.com/supabase/supabase-flutter/issues/2 RealtimeClient.push.callback. (package:realtime_client/src/realtime_client.dart:293) https://github.com/supabase/supabase-flutter/issues/3 new RealtimeClient. (package:realtime_client/src/realtime_client.dart:137) https://github.com/supabase/supabase-flutter/pull/4 RealtimeClient.push.callback (package:realtime_client/src/realtime_client.dart:293) https://github.com/supabase/supabase-flutter/issues/5 RealtimeClient.push (package:realtime_client/src/realtime_client.dart:310) https://github.com/supabase/supabase-flutter/pull/6 Push.send (package:realtime_client/src/push.dart:61) https://github.com/supabase/supabase-flutter/issues/7 Push.resend (package:realtime_client/src/push.dart:52) https://github.com/supabase/supabase-flutter/issues/8 RealtimeChannel.rejoin (package:realtime_client/src/realtime_channel.dart:563) https://github.com/supabase/supabase-flutter/issues/9 RealtimeChannel.rejoinUntilConnected (package:realtime_client/src/realtime_channel.dart:189) https://github.com/supabase/supabase-flutter/issues/10 new RealtimeChannel. (package:realtime_client/src/realtime_channel.dart:142)

[maxfornacon]

Leaving some thoughts for discussion:

When subscribing to a realtime channel you can pass a callback function to the subscribe method like this:

.subscribe((status, [error]) {
  if (status == RealtimeSubscribeStatus.closed) {
    print('!!!!!! CLOSED !!!!!!!');
  }
});

Would it be possible to add an optional callback function to the stream method as well? This would not fix the primary issue that the channel is being closed but it would allow us to add some custom functionality to the client. I could imagine showing the users some info that the stream is disconnected and possibly not showing the latest data or even adding some logic to reinitialise the stream automatically once it is closed.

Another approach could be exposing the channel name (topic) from the SupabaseStreamBuilder and then subscribing to the channel and adding a callback function there to listen for the closed status.

[pseacrest]

In my case, I noticed that when the app is minimized or out of focus for an extended time, the connection to the server often drops silently. Like you, there are no obvious exceptions or errors that would indicate a failure, which makes troubleshooting tricky. I’ve also been investigating whether the access token refresh could be a factor here, especially if the app is unable to properly reauthenticate when the token expires. I’ve also thought that a lack of heartbeat signals could be causing the server to close connections prematurely.

From my research so far, a few things stand out:

1. Token Refresh Issues: As you mentioned, setting a short token expiry time does seem to exacerbate the problem. I haven’t dug deep into the token refresh flow yet, but it seems like a connection might be silently dropped if the token refresh doesn’t propagate correctly to all channels. I’m also curious if the token refresh mechanism has a race condition when multiple channels are active, as I noticed more problems when running multiple streams simultaneously.
2. Backgrounding the App: I’ve had streams close unexpectedly when the app is backgrounded, even though it shouldn’t be inactive in terms of network connections. Your mention of the AppLifecycleState.hidden behavior sounds like a plausible connection. I’ve been exploring keeping the app “alive” in the background, but that’s more of a workaround than a real solution.
3. Handling Channel Closures: I totally agree that adding a callback function for the StreamBuilder would be useful for detecting when streams are closed. In my project, I implemented a manual reconnect mechanism based on the closed event, but I believe having a cleaner solution like a callback within StreamBuilder would be ideal. As you mentioned, even just a notification to users when the stream closes would be helpful.

[maxfornacon]

In my project, I implemented a manual reconnect mechanism based on the closed event, but I believe having a cleaner solution like a callback within StreamBuilder would be ideal.

@pseacrest Could you go into more detail on how exactly you did it?

But let's see if the PR @Vinzent03 did fixes the issues. Maybe he can leave a comment on when the work will be finished.

[Vinzent03]

With #1019 being merged now, these issues should be solved now. You can try them by upgrading supabase_flutter to 2.7.0. If you still experience any issues, please create a new issue.