zino-hofmann / graphql-flutter

A GraphQL client for Flutter, bringing all the features from a modern GraphQL client to one easy to use package.
https://zino-hofmann.github.io/graphql-flutter
MIT License
3.25k stars 620 forks source link

Websocket connection does not reconnect after WebSocketChannelException #1253

Open orestesgaolin opened 1 year ago

orestesgaolin commented 1 year ago

Describe the issue Connection drops with the exception WebSocketChannelException and cannot be restored without restarting the app. The problem occurs more or less regularly when the app is idle for at least 20-30 minutes e.g. in the background.

To Reproduce

I couldn't find an easy way to reproduce it, but I will try with the example app.

I'm using custom WebSocketLink that uses compute to parse messages and maybe I'm missing something that was already fixed?

class CustomWsLink extends WebSocketLink {
  CustomWsLink(
    super.url, {
    super.config,
    super.subProtocol,
    required this.getToken,
    required this.userUid,
  });

  final Future<String?> Function() getToken;
  final String userUid;
  _Connection? _connection;

  /// this will be called every time you make a subscription
  @override
  // ignore: type_annotate_public_apis
  Stream<Response> request(Request request, [forward]) async* {
    final token = await getToken();
    if (token == null) {
      throw Exception('Auth token is missing');
    }

    /// check is connection is null or the token changed
    if (_connection == null || _connection!.token != token) {
      _connectOrReconnect(token);
    }
    yield* _connection!.client.subscribe(request, true);
  }

  /// Connects or reconnects to the server with the specified headers.
  void _connectOrReconnect(String token) {
    _connection?.client.dispose();

    _connection = _Connection(
      client: SocketClient(
        url,
        config: SocketClientConfig(
          autoReconnect: true,
          inactivityTimeout: const Duration(seconds: 30),
          queryAndMutationTimeout: const Duration(seconds: 10),
          delayBetweenReconnectionAttempts: const Duration(seconds: 5),
          connectFn: (uri, protocols) async {
            await getToken();
            var channel = WebSocketChannel.connect(uri, protocols: protocols);
            // ignore: join_return_with_assignment
            channel = channel.forGraphQLIsolate();

            return channel;
          },
          initialPayload: () async {
            return {
              'headers': {
                'Authorization': await getToken(),
                'X-Hasura-User-Id': userUid,
                'X-Hasura-Role': 'user',
                'Content-Type': 'application/json',
              },
            };
          },
        ),
      ),
      token: token,
    );
  }

  @override
  Future<void> dispose() async {
    await _connection?.client.dispose();
    _connection = null;
  }
}

/// this a wrapper for web socket to hold the used token
class _Connection {
  _Connection({
    required this.client,
    required this.token,
  });

  final SocketClient client;
  final String token;
}

class CustomGraphQLWebSocketChannel extends GraphQLWebSocketChannel {
  CustomGraphQLWebSocketChannel(super.webSocket);

  Stream<GraphQLSocketMessage>? _messages;

  /// Stream of messages from the endpoint parsed as GraphQLSocketMessages
  @override
  Stream<GraphQLSocketMessage> get messages =>
      _messages ??= stream.asyncMap<GraphQLSocketMessage>(
        (d) async => compute(safeParse, d),
      );
}

GraphQLSocketMessage safeParse(dynamic message) {
  try {
    return GraphQLSocketMessage.parse(message);
  } catch (e) {
    if (kDebugMode) {
      print('Error parsing message $message\n$e');
    }
    return ConnectionError(message);
  }
}

extension GraphQLGetter on WebSocketChannel {
  /// Returns a wrapper that has safety and convenience features for graphql
  GraphQLWebSocketChannel forGraphQLIsolate() =>
      this is CustomGraphQLWebSocketChannel
          ? this as CustomGraphQLWebSocketChannel
          : CustomGraphQLWebSocketChannel(this);
}

Expected behavior

The websocket eventually reconnects

device / execution context Happens both on iOS and Android

Other useful/optional fields

Stacktrace: ```dart #0 new IOWebSocketChannel._withoutSocket. #1 Stream.handleError. (dart:async/stream.dart:929:16) #2 _HandleErrorStream._handleError (dart:async/stream_pipe.dart:269:17) #3 _ForwardingStreamSubscription._handleError (dart:async/stream_pipe.dart:157:13) #4 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) #5 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) #6 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) #7 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) #8 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) #9 _StreamController._addError (dart:async/stream_controller.dart:656:7) #10 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) #11 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) #12 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) #13 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) #14 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) #15 _StreamController._addError (dart:async/stream_controller.dart:656:7) #16 new Stream.fromFuture. (dart:async/stream.dart:251:18) #17 _RootZone.runBinary (dart:async/zone.dart:1658:54) #18 _FutureListener.handleError (dart:async/future_impl.dart:162:22) #19 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47) #20 Future._propagateToListeners (dart:async/future_impl.dart:799:13) #21 Future._completeError (dart:async/future_impl.dart:574:5) #22 Future._asyncCompleteError. (dart:async/future_impl.dart:665:7) #23 _microtaskLoop (dart:async/schedule_microtask.dart:40:21) #24 _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5) ```
Logs: ``` I/flutter (31184): Disconnected from websocket. [log] [DataClient] πŸ’‘ Token expires at 2022-11-02 17:55:33.000 I/flutter (31184): Initialising connection [log] [DataClient] πŸ’‘ Token expires at 2022-11-02 17:55:33.000 I/flutter (31184): [SocketClient] message stream encountered error: WebSocketChannelException: WebSocketChannelException: HttpException: Connection reset by peer, uri = https://redacted.hasura.app:0/v1/graphql I/flutter (31184): stacktrace: I/flutter (31184): #0 new IOWebSocketChannel._withoutSocket. I/flutter (31184): #1 Stream.handleError. (dart:async/stream.dart:929:16) I/flutter (31184): #2 _HandleErrorStream._handleError (dart:async/stream_pipe.dart:269:17) I/flutter (31184): #3 _ForwardingStreamSubscription._handleError (dart:async/stream_pipe.dart:157:13) I/flutter (31184): #4 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) I/flutter (31184): #5 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) I/flutter (31184): #6 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) I/flutter (31184): #7 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) I/flutter (31184): #8 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) I/flutter (31184): #9 _StreamController._addError (dar E/flutter (31184): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: WebSocketChannelException: WebSocketChannelException: HttpException: Connection reset by peer, uri = https://redacted.hasura.app:0/v1/graphql E/flutter (31184): #0 new IOWebSocketChannel._withoutSocket. E/flutter (31184): #1 Stream.handleError. (dart:async/stream.dart:929:16) E/flutter (31184): #2 _HandleErrorStream._handleError (dart:async/stream_pipe.dart:269:17) E/flutter (31184): #3 _ForwardingStreamSubscription._handleError (dart:async/stream_pipe.dart:157:13) E/flutter (31184): #4 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) E/flutter (31184): #5 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) E/flutter (31184): #6 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) E/flutter (31184): #7 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) E/flutter (31184): #8 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) E/flutter (31184): #9 _StreamController._addError (dart:async/stream_controller.dart:656:7) E/flutter (31184): #10 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) E/flutter (31184): #11 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) E/flutter (31184): #12 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) E/flutter (31184): #13 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) E/flutter (31184): #14 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) E/flutter (31184): #15 _StreamController._addError (dart:async/stream_controller.dart:656:7) E/flutter (31184): #16 new Stream.fromFuture. (dart:async/stream.dart:251:18) E/flutter (31184): #17 _RootZone.runBinary (dart:async/zone.dart:1658:54) E/flutter (31184): #18 _FutureListener.handleError (dart:async/future_impl.dart:162:22) E/flutter (31184): #19 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47) E/flutter (31184): #20 Future._propagateToListeners (dart:async/future_impl.dart:799:13) E/flutter (31184): #21 Future._completeError (dart:async/future_impl.dart:574:5) E/flutter (31184): #22 Future._asyncCompleteError. (dart:async/future_impl.dart:665:7) E/flutter (31184): #23 _microtaskLoop (dart:async/schedule_microtask.dart:40:21) E/flutter (31184): #24 _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5) E/flutter (31184): E/flutter (31184): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: WebSocketChannelException: WebSocketChannelException: HttpException: Connection reset by peer, uri = https://redacted.hasura.app:0/v1/graphql E/flutter (31184): #0 new IOWebSocketChannel._withoutSocket. E/flutter (31184): #1 Stream.handleError. (dart:async/stream.dart:929:16) E/flutter (31184): #2 _HandleErrorStream._handleError (dart:async/stream_pipe.dart:269:17) E/flutter (31184): #3 _ForwardingStreamSubscription._handleError (dart:async/stream_pipe.dart:157:13) E/flutter (31184): #4 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) E/flutter (31184): #5 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) E/flutter (31184): #6 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) E/flutter (31184): #7 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) E/flutter (31184): #8 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) E/flutter (31184): #9 _StreamController._addError (dart:async/stream_controller.dart:656:7) E/flutter (31184): #10 _RootZone.runBinaryGuarded (dart:async/zone.dart:1598:10) E/flutter (31184): #11 _BufferingStreamSubscription._sendError.sendError (dart:async/stream_impl.dart:358:15) E/flutter (31184): #12 _BufferingStreamSubscription._sendError (dart:async/stream_impl.dart:376:7) E/flutter (31184): #13 _BufferingStreamSubscription._addError (dart:async/stream_impl.dart:280:7) E/flutter (31184): #14 _SyncStreamControllerDispatch._sendError (dart:async/stream_controller.dart:778:19) E/flutter (31184): #15 _StreamController._addError (dart:async/stream_controller.dart:656:7) E/flutter (31184): #16 new Stream.fromFuture. (dart:async/stream.dart:251:18) E/flutter (31184): #17 _RootZone.runBinary (dart:async/zone.dart:1658:54) E/flutter (31184): #18 _FutureListener.handleError (dart:async/future_impl.dart:162:22) E/flutter (31184): #19 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47) E/flutter (31184): #20 Future._propagateToListeners (dart:async/future_impl.dart:799:13) E/flutter (31184): #21 Future._completeError (dart:async/future_impl.dart:574:5) E/flutter (31184): #22 Future._asyncCompleteError. (dart:async/future_impl.dart:665:7) E/flutter (31184): #23 _microtaskLoop (dart:async/schedule_microtask.dart:40:21) E/flutter (31184): #24 _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5) ``` and then few more times the same stacktrace

additional context

Dependencies:

|-- data_client 1.0.0
|   |-- equatable...
|   |-- firebase_auth...
|   |-- flutter...
|   |-- graphql 5.1.2-beta.5
|   |   |-- collection...
|   |   |-- gql...
|   |   |-- gql_dedupe_link 2.0.3
|   |   |   |-- async...
|   |   |   |-- gql_exec...
|   |   |   |-- gql_link...
|   |   |   '-- meta...
|   |   |-- gql_error_link 0.2.3
|   |   |   |-- async...
|   |   |   |-- gql_exec...
|   |   |   |-- gql_link...
|   |   |   '-- meta...
|   |   |-- gql_exec 0.4.1
|   |   |   |-- collection...
|   |   |   |-- gql...
|   |   |   '-- meta...
|   |   |-- gql_http_link 0.4.4+1
|   |   |   |-- gql...
|   |   |   |-- gql_exec...
|   |   |   |-- gql_link...
|   |   |   |-- http...
|   |   |   |-- http_parser...
|   |   |   '-- meta...
|   |   |-- gql_link 0.5.1-alpha+1660256327632
|   |   |   |-- gql...
|   |   |   |-- gql_exec...
|   |   |   '-- meta...
|   |   |-- gql_transform_link 0.2.3-alpha+1660256327655
|   |   |   |-- gql_exec...
|   |   |   '-- gql_link...
|   |   |-- hive 2.2.3
|   |   |   |-- crypto...
|   |   |   '-- meta...
|   |   |-- http...
|   |   |-- meta...
|   |   |-- normalize 0.7.2
|   |   |   |-- collection...
|   |   |   '-- gql...
|   |   |-- path...
|   |   |-- rxdart...
|   |   |-- stream_channel...
|   |   |-- uuid 3.0.6
|   |   |   '-- crypto...
|   |   '-- web_socket_channel 2.2.0
|   |       |-- async...
|   |       |-- crypto...
|   |       '-- stream_channel...
|   |-- http...
|   |-- intl...
|   |-- json_annotation...
|   |-- log...
|   |-- rxdart...
|   '-- simple_date...

Flutter doctor

Doctor summary (to see all details, run flutter doctor -v):
[βœ“] Flutter (Channel stable, 3.3.6, on macOS 12.6 21G115 darwin-arm, locale pl-PL)
[!] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    βœ— cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    βœ— Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[βœ“] Xcode - develop for iOS and macOS (Xcode 14.1)
[βœ“] Chrome - develop for the web
[βœ“] Android Studio (version 2021.2)
[βœ“] VS Code (version 1.72.2)
[βœ“] Connected device (4 available)
[βœ“] HTTP Host Availability
orestesgaolin commented 1 year ago

I was trying to reproduce the problem on the starwars example app, but it works just fine with that:

  1. Start the example starwars server locally and run the example starwars app on Android emulator
  2. Turn airplane mode on the emulator
  3. Websocket gets disconnected
  4. You can see I/flutter ( 5978): There was an error causing connection lost: SocketException: Connection failed (OS Error: Network is unreachable, errno = 101), address = 10.0.2.2, port = 3000
  5. But after turning on the internet the connection restores with I/flutter ( 5978): Initialising connection
CleanShot 2022-11-08 at 22 13 54@2x

In case of my CustomWsLink I was able to capture the exception in onStreamError of SocketClient and reconnect as follows. When the WebSocketChannelException is thrown, the onStreamError is called and this is how I can react to this by repeatedly trying to reconnect. It's a proof of concept, but for now it seems to work better than without it.

  void _connectOrReconnect(String token) {
    _connection?.client.dispose();

    _connection = _Connection(
      client: SocketClient(
        url,
        // the default implementation does nothing with the exception
        onStreamError: (error, stackTrace) {
          Logger(tag: 'CustomWsLink').e('Socket error:', error, stackTrace);
          Future<void>.delayed(const Duration(milliseconds: 1500))
              .then((value) {
            _connectOrReconnect(token);
          });
        },
        config: SocketClientConfig(
          autoReconnect: true,
          inactivityTimeout: const Duration(seconds: 30),
          queryAndMutationTimeout: const Duration(seconds: 10),
          delayBetweenReconnectionAttempts: const Duration(seconds: 5),
          connectFn: (uri, protocols) async {
            await getToken();
            var channel = WebSocketChannel.connect(uri, protocols: protocols);
            // ignore: join_return_with_assignment
            channel = channel.forGraphQLIsolate();

            return channel;
          },
          initialPayload: () async {
            return {
              'headers': {
                'Authorization': await getToken(),
                'X-Hasura-User-Id': userUid,
                'X-Hasura-Role': 'user',
                'Content-Type': 'application/json',
              },
            };
          },
        ),
      ),
      token: token,
    );
  }
vincenzopalazzo commented 1 year ago

Sorry to be late there, I will try to reproduce it in some of the new demo

GoncaloPT commented 1 year ago

From some digging I'm doing ( facing the same problem, errors on the stream are swallowed along the way ) seems the problem is the SocketClient which assumes a default onStreamError which only prints errors. My idea would be to create with a custom websocketlink which passes a streamError function which ( depending on the type of error ) will trigger the reconnect. More on that later.

GoncaloPT commented 1 year ago

I can confirm it works. ( after better reading your comments i've seen that you've reached the same conclusions ).

The fix would be to do something like (on WebSocketLink)

@override
  void connectOrReconnect() {
    _socketClient?.dispose();
    _socketClient = graphql.SocketClient(url,
        config: config,
        protocol: subProtocol,
        onStreamError: _reconnectOnNetworkError);
  }
_reconnectOnNetworkError(err, stack) {
    if (err is NetworkException &&
        err.type == NetworkExceptionType.connectionFailure) {
      connectOrReconnect();
      return;
    }
    // we have no way of capturing the stream here. if we do nothing the
    // error is just swallowed
    throw err;
  }

This NetworkException is a custom exception that I use on my custom WebSocketChannel, but any other mechanism can be used to detect if it is a recoverable exception or not. Maybe it would make sense to expose a way for users to provide a predicate function where they can decide what is considered to be a recoverable error.