[subscriptions] http sets content-length = 0 breaking some providers i.e. google cloud run

[brianschardt]

Describe the issue When I try to use a subscription it goes in an infinite loop between connect and disconnect, and as a result, does not work on cloud environment. What is weird is it works against my local backend but not my cloud gcp cloud run api. However, I have tested my cloud backend api with other graphql subscription clients and it works perfectly. So I know my backend is not the issue.

Expected behavior Stay connected

device / execution context IOS Simulator

Other useful/optional fields

screenshots

additional context Graphql, Apollo Server, NestJs

[micimize]

To quote myself from an earlier issue:

the websocket link automatically disconnects on inactivity timeout, and reconnects if the autoreconnect config is set. See discussion in #653 for more info

SocketClientConfig(
  autoReconnect: true, // false to disable reconnect
  inactivityTimeout: Duration(seconds: 30), // set to 0 to disable timeout
)

Do actual client.subscribe or Subscription calls yield any results? If there is an issue I'll need some actual code to diagnose it

[brianschardt]

Here is a quick short code test project that shows my issue

import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:graphql/client.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  init() {
    WebSocketLink wsLink =
        WebSocketLink('wss://api.dev.iris.finance/subscription',
            config: SocketClientConfig(
              autoReconnect: true, // false to disable reconnect
              inactivityTimeout:
                  Duration(seconds: 0), // set to 0 to disable timeout
            ));
    GraphQLClient client = GraphQLClient(
      cache: GraphQLCache(),
      link: wsLink,
    );

    final subscriptionDocument = gql(
      r'''
subscription {
  collectionAddedText(input: {collectionKey: 1}) {
    text {
      textKey
      createdAt
      orderedCreatedAt
      value
    }
  }
}
  ''',
    );

    var s = client.subscribe(
      SubscriptionOptions(document: subscriptionDocument),
    );
    s.listen((d) {
      print('RECEIVED');
    });
  }

  @override
  Widget build(BuildContext context) {
    init();
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: Container(
        color: Colors.blue,
      ),
    );
  }
}

[brianschardt]

Also to note it connects and then disconnects immediately. I have played around with changing these values

inactivityTimeout: Duration(seconds: 0), // set to 0 to disable timeout

[Setti7]

Cloud run had problems with websockets in the past, maybe its a problem on their end?

[micimize]

My guess is that this has something to do with their authentication scheme.

However, I have tested my cloud backend api with other graphql subscription clients and it works perfectly

If you provide all the details of networks requests made by the other clients (headers, etc) we can probably figure it out. May have to use a header-enabled websocket link

[brianschardt]

I got the headers from nodejs client code that works:

GET /subscription HTTP/1.1
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: LchF/g+2UX0Z3X86VobTyg==
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: graphql-ws

Here is nodejs client code.

const WebSocket = require('ws');

let wsUrl = 'wss://api.dev.iris.finance/subscription';

const ws = new WebSocket(wsUrl, ['graphql-ws']);
let message = '{"id":"1","type":"start","payload":{"variables":{},"extensions":{},"operationName":null,"query":"subscription {  collectionAddedText(input: {collectionKey: 4}) {    text {      textKey      value    }  }}"}}'

ws.on('open', function open() {
  console.log('Open; sending message');
  ws.send(message);
});

ws.on('message', function incoming(data) {
  console.log('Receive data');
  console.log(data);
});

ws.on('close', function close(d) {
  console.log('disconnected d', d);
});

[brianschardt]

I think this is completely separate from this graphql package and think this is a flutter/dart issue.

[micimize]

@brianschardt I tried both your dart code and node code, given that. The dart code (simplified with some logging) disconnects with a 1005, "No Status Received" while the node code just hangs after sending a message.

I highly doubt there is an issue with dart or the underlying websocket implementation.

import 'package:graphql/client.dart';

void main() {
  final wsLink = WebSocketLink(
    'wss://api.dev.iris.finance/subscription',
    config: SocketClientConfig(
      autoReconnect: true, // false to disable reconnect
      // 0 second inactivityTimeout doesn't disable it
      inactivityTimeout: const Duration(seconds: 30),
      onConnectOrReconnect: (ws) => ws.stream.listen(
        (event) {
          print(['data', event]);
        },
        onError: (event) {
          print(['error', event]);
        },
        onDone: () {
          print(['done', ws.closeCode, ws.closeReason]);
        },
      ),
    ),
  );
  final client = GraphQLClient(
    cache: GraphQLCache(),
    link: wsLink,
  );

  final subscriptionDocument = gql(
    r'''
subscription {
  collectionAddedText(input: {collectionKey: 4}) {
    text {
      textKey
      createdAt
      orderedCreatedAt
      value
    }
  }
}
  ''',
  );

  var s = client.subscribe(
    SubscriptionOptions(document: subscriptionDocument),
  );
  s.listen((d) {
    print('RECEIVED');
  });
}

[brianschardt]

@micimize Because this is a websocket hanging is expected, it is listening for data. We actually figured out the issue, its deep into the dart websocket http library. They automatically set content-length = 0 in the headers. Once we remove that it all works as expected.

[micimize]

right you are – we should integrate the workaround in https://github.com/dart-lang/sdk/issues/43574#issuecomment-761106329 into the library

[brianschardt]

@micimize i agree is this something that you can do easily? If not ill look into having my team implement it.

[micimize]

@brianschardt I am fairly busy these days, so I'd recommend PRing to beta if you want it merged urgently.

[thomas-ferchau]

I suggest a modification to websocket_impl.dart if changing http_impl.dart is too wide-ranging. The proposed change can be seen here: https://github.com/dart-lang/sdk/issues/45139#issuecomment-828344653

There seems to be another problem in case of a redirect from HTTP to HTTPS and then upgrading to WSS: https://github.com/dart-lang/sdk/issues/43574#issuecomment-828366042

[hschk]

@micimize This issue still seems to exist. When I observe the request headers, a content-length: 0 header is sent, which causes it to reconnect continuously with Cloud Run. Is there at least a way to remove the content-length header manually? Thanks for your help!

[micimize]

Hi @hschk — I haven't been a maintainer of this project for some time, but if things haven't changed dramatically you can probably go into the gql link middleware for subscriptions and patch the header issue.

The architecture doc should help point you in the right direction