Handle long-polling requests

lwouis commented 5 years ago

Description

I'm using Firestore in my project. Firestore seems to use long-polling on HTTP GET requests to provide its real-time updates. This means firestore js library (i'm using angularfire2) first makes a POST request to firestore servers. That one is fine. Then it does a GET request, which is kept open for a long time, around a minute. If you look at that GET request in Chrome's devtool, it will show you an empty response because devtools are waiting for the request to close I guess to display the response. However the js code already has access to the chunks of data, so firebase already displays it on the page.

When polly.js gets in the picture, even in passthrough mode, it crashes. The POST request works fine, but the subsequent long-lived GET queries, even if passed-through, are breaking firebase js call-site.

The fact that passthrough also breaks firestore means it's a no-go for my project. I'm really eager to use polly.js, and I would love to find a workaround even if it means I can ignore those firebase calls and mock the other calls. Best would be of course to be able to mock those long-polling calls as well.

Error Message & Stack Trace

In order:

Passthrough ➞ POST https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fdrone-advisor-test%2Fdatabases%2F%28default%29&VER=8&RID=12971&CVER=22&X-HTTP-Session-Id=gsessionid&%24httpHeaders=X-Goog-Api-Client%3Agl-js%2F%20fire%2F6.3.1%0D%0A&zx=9db2uev7uvsd&t=1 200 • 26ms

At this point, the GET request is intercepted by polly.js, but polly is not logging it cause the request is still not closed.

After 10s, firebase call-site which is wrapped by polly fails to get the data they want, so they print this:

index.esm.js:117 [2019-07-26T05:13:21.884Z]  @firebase/firestore: Firestore (6.3.1): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

After more time, like 1min, the request is closed (I assume by the server), and polly prints:

pollyjs-core.js:7152 Passthrough ➞ GET https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fdrone-advisor-test%2Fdatabases%2F%28default%29&gsessionid=Y97JxkYuiOdaHZaj8QPKyNK5tVncDx6m&VER=8&RID=rpc&SID=okOtkYysixnek6sHNypRCQ&CI=0&AID=0&TYPE=xmlhttp&zx=gb8xvyqqxewm&t=1 200 • 47174ms
pollyjs-core.js:7163 Errored ➞ GET https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fdrone-advisor-test%2Fdatabases%2F%28default%29&gsessionid=Y97JxkYuiOdaHZaj8QPKyNK5tVncDx6m&VER=8&RID=rpc&SID=okOtkYysixnek6sHNypRCQ&CI=0&AID=0&TYPE=xmlhttp&zx=gb8xvyqqxewm&t=1
pollyjs-core.js:7164 Error: INVALID_STATE_ERR - 0
    at verifyRequestOpened (pollyjs-adapter-xhr.js:6788) [<root>]
    at FakeXMLHttpRequest.setStatus (pollyjs-adapter-xhr.js:7028) [<root>]
    at FakeXMLHttpRequest.respond (pollyjs-adapter-xhr.js:7193) [<root>]
    at XHRAdapter.respondToRequest (pollyjs-adapter-xhr.js:11071) [<root>]
    at XHRAdapter._callee14$ (pollyjs-adapter-xhr.js:10859) [<root>]
    at tryCatch (pollyjs-adapter-xhr.js:68) [<root>]
    at Generator.invoke [as _invoke] (pollyjs-adapter-xhr.js:294) [<root>]
    at Generator.prototype.<computed> [as next] (pollyjs-adapter-xhr.js:120) [<root>]
    at asyncGeneratorStep (pollyjs-adapter-xhr.js:1961) [<root>]
    at _next (pollyjs-adapter-xhr.js:1983) [<root>]
    at :8100/vendor.js:129842:19 [<root>]
    at new Promise (pollyjs-adapter-xhr.js:1819) [<root>]
    at XHRAdapter.<anonymous> (pollyjs-adapter-xhr.js:1979) [<root>]
    at XHRAdapter.onRequestFinished (pollyjs-adapter-xhr.js:10873) [<root>]

I think this class may be the firestore network implementation.

Config

Copy the config used to setup the Polly instance:

this.polly = new Polly('test-recording', {
      adapters: [adapterFetch, adapterXhr],
      persister: persisterLocalStorage,
      matchRequestsBy: {
        headers: false,
        url: {
          query: false,
        },
      },
      logging: true,
})
const {server} = this.polly
server.any(['/*', 'https://firestore.googleapis.com/*']).passthrough()

Dependencies

Copy the @pollyjs dependencies from package.json:

{
   "@pollyjs/adapter-fetch": "^2.6.0",
   "@pollyjs/adapter-xhr": "^2.6.0",
   "@pollyjs/core": "^2.6.0",
   "@pollyjs/persister-local-storage": "^2.6.0"
}

Environment

This is running in-browser

chrome Version 75.0.3770.142 (Official Build) (64-bit)
macOS 10.13.6 (17G7024)

jasonmit commented 5 years ago

To avoid us having to try and replicate your project's setup and/or go through the motions of setting up firebase, can you share with us the project or create a reproduction of the issue?

This will speed up our ability to triage and patch.

I would love to find a workaround even if it means I can ignore those firebase calls and mock the other calls

In the meantime, have you tried to intercept the firestore/firebase calls and mock a response? i.e.,

polly
  .server
    .get('https://firestore.googleapis.com/*')
    .intercept((req, res) => res.sendStatus(200));

lwouis commented 5 years ago

@jasonmit I made a public repo to reproduce. Just git clone it and run npm start to witness the issue. If you comment out lines 22-34 in app.component.ts which correspond to Pollyjs instanciation, you will see the correct behaviour

jasonmit commented 5 years ago

It seems there are many layers involved.

zone.js (required by angular) is extending the original, native, XHR object and this is done before Polly replaces the global XHR object.

After fixing that, I noticed firestore aborts the XHR requests after 30s and creates a new request. However, when dealing with Polly the fakeXhr instances have a readyState of 1 ("opened") at the time they're being aborted. With Polly disable, the native XHR instances have a readyState of 4 ("done") at the time abort is called. I'd like to figure out why this is, should help us understand what needs patched.

I'll need to circle back once I have more cycles to devote again.

Netflix / pollyjs