Open mikelehen opened 5 years ago
@wenbozhu Browser: Chrome 86.0.4240.75 OS: Windows 10 Bullguard Premium Version: 21.0.385.9 with Safe Browsing feature turned on (this would be on by default)
Let me know if you need any further details.
@mikelehen
What environment causes the problem (app platform, antivirus software, network proxy, network conditions, etc.)? React-Native / Windows defender / No proxy / Network conditions: excellent. "dependencies": { "@react-native-community/masked-view": "^0.1.10", "@react-navigation/native": "^5.8.6", "@react-navigation/stack": "^5.12.4", "axios": "^0.21.0", "firebase": "^8.0.1", "react": "16.13.1", "react-native": "0.63.3", "react-native-gesture-handler": "^1.8.0", "react-native-paper": "^4.3.1", "react-native-reanimated": "^1.13.1", "react-native-safe-area-context": "^3.1.8", "react-native-screens": "^2.13.0", "react-native-vector-icons": "^7.1.0" } What is the behavior without experimentalForceLongPolling? Getting this error @firebase/firestore: Firestore (8.0.1): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds. This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.
Does experimentalForceLongPolling completely resolve the issue? Yes, it seems so. Although the data is still fetched almost immediately (definitely in under 10s), and yet the call fails when I am not using this setting.
Please visit https://debug-my.firebaseapp.com/, wait for "All tests done" (about 60 seconds), paste the results into a gist, and paste the link in your comment. https://gist.github.com/shahyash62/4c90c412309de5673d9b3e15046b608a
HTTP proxies, or network hardware, all that's trying to evaluate the whole payload of an HTTP response before allowing any of it through. Firestore's default mechanism for pushing updates from the server sends multiple logical updates in the same HTTP response. What
experimentalForceLongPolling
does is to restrict the server from sending only a single update per HTTP response. The server then waits until it gets another request from the client to send the next update.The performance penalty falls out of that description: it's one round trip per update. On fast networks proximate to the Firestore instance, this isn't something you'll notice. On slow networks or configurations traveling a long distance to the server it's more of an issue.
As noted elsewhere, we're working on adding code that auto-detects this condition.
Oh I understand. My apps doesn`t work with Samsung J6, and it connects only with 2.4Ghz Wi-fi.
With experimentalAutoDetectLongPolling I'm getting lots of couldn't reach cloud firestore in 10 seconds errors. With the flag disabled everything works fine, so it looks like the flag influences the standard mode.
@rafikhan thanks for the update. The easiest way I reproduce the issue and test is with BullGuard antivirus. I just upgraded to firebase 7.24.0 and tested my app. My finding are the following,
With experimentalAutoDetectLongPolling flag on: I no longer see the error message "unable to reach firebase backend" on the browser console. However, the app still takes close to a minute to boot up.
With experimentalForceLongPolling flag on: No errors like the previous flag and app boots up with in a matter of seconds.
So I guess the new flag is in the right direction but still there is scope for further optimization.
Thanks!
From the gist you posted in this comment the very first POST request of webchannel failed to get any response and was terminated after 20s..we don't know the reason though. However, such kind of logs can be very helpful for us to debug. Do you have similar logs for experimentalForceLongPolling mode and experimentalAutoDetectLongPolling mode?
Just encountered the exact same problem on my PC using experimentalAutoDetectLongPolling: true here is the debug gist: https://gist.github.com/chenlevy/af308e5fd288ba383abd43110a3e5d25 tell me if there is anything else I can do.
I have some users that are experiencing this issue. A user gets the following error on my site:
And with the following result on your test site:
With experimentalForceLongPolling flag on: Unreliable/slow connection. But the user is able to access the site
With experimentalAutoDetectLongPolling flag on: Mostly good connection but sometimes the connection fails with the following error:
It seems like these options are not stable enough yet, and my only option at this point seems to be to ignore the users not able to visit the site.
Update: The connection error with experimentalAutoDetectLongPolling turned on seems to occur when multiple tabs are open. When I close other tabs and refresh the page, the connection works.
Just encountered the exact same problem on my PC using experimentalAutoDetectLongPolling: true here is the debug gist: https://gist.github.com/chenlevy/af308e5fd288ba383abd43110a3e5d25 tell me if there is anything else I can do.
Hi! I'm not from Firebase team, but I can help you with the webchannel stuff(the underlying connection) here.
For the first three tests of this gist, the very first POST request of webchannel failed. It's more like something wrong with the testing connection https://webchannel.googleapis.com, https://webchannel.sandbox.google.com, because from my knowledge of webchannel, such kind of failure is unrelated to longpolling... And if my understanding is correct, the flag 'experimentalAutoDetectLongPolling' is not tested in those tests, right?
I suppose https://webchannel.googleapis.com, etc are testing connections for debugging provided by Firebase? Did you get the error message using 'experimentalAutoDetectLongPolling' with this testing connection, or the connection of your own apps?
We're seeing significantly delays in the initial connection when enabling experimentalAutoDetectLongPolling
when running in a Gitlab CI/CD Docker container. Hard to say how long but probably on the order of a minute or so.
There are no delays when using the force long polling option or not setting anything at all.
Just encountered the exact same problem on my PC using experimentalAutoDetectLongPolling: true here is the debug gist: https://gist.github.com/chenlevy/af308e5fd288ba383abd43110a3e5d25 tell me if there is anything else I can do.
Hi! I'm not from Firebase team, but I can help you with the webchannel stuff(the underlying connection) here.
For the first three tests of this gist, the very first POST request of webchannel failed. It's more like something wrong with the testing connection https://webchannel.googleapis.com, https://webchannel.sandbox.google.com, because from my knowledge of webchannel, such kind of failure is unrelated to longpolling... And if my understanding is correct, the flag 'experimentalAutoDetectLongPolling' is not tested in those tests, right?
I suppose https://webchannel.googleapis.com, etc are testing connections for debugging provided by Firebase? Did you get the error message using 'experimentalAutoDetectLongPolling' with this testing connection, or the connection of your own apps?
Thanks for your reply, actually the 'experimentalAutoDetectLongPolling: true' flag was tested in those tests. I was able to reproduce it on my dev machine occasionally and some of my users are also experiencing this issue. Any news about this? I'm thinking about completely remove any direct calls to firebase in my code and to access the data through api. It'll take a lot of effort so I prefer not to do so, Does the Firebase team has any news about this issue or how to solve/workaround it?
Update from Firebase Hi everyone, thanks for being patient. This issue is definitely a priority and we have been internally discussing how to move it forward. As of right now we need more detailed logging from the networking library we use to talk to the backend (WebChannel).
To get this data we'd like to ask those of you experiencing the one minute delay when using experimentalAutoDetectLongPolling
to deploy a special Firebase SDK that automatically logs detailed network traffic and share the contents back with us. You can download it via npm install firebase@longpoll
which includes Firebase 8.2.2.
Please give this a try. Thank you.
Tag: @scottfr @chenlevy @pimmee @Benny739 @catchshyam
I'm also struggling with this issue in one of our larger apps, and specifically am running into trouble with folks that have the parent control software from qustodio.com installed on their computer. For my users, even if they have Qustodio set to allow all traffic through, they run into this "Backend didn't respond within 10 seconds" error when trying to access our app. If they instead completely turn off Qustodio, then everything works fine. We tried using the "experimentalAutoDetectLongPolling" option, which did not resolve the issue. If we instead use the "experimentalForceLongPolling" option it does resolve the issue, but we'd prefer not to make 'long polling' the default approach for all of our users due to the performance trade offs.
We'll see if we can enable in our app the special SDK requested in the prior post, but I wanted to share in the meantime that installing Qustodio is an easy way to consistently reproduce at least one use case where the "experimentalAutoDetectLongPolling" option doesn't seem to do the trick.
Thank you for your efforts in this! I'm one of the many big Firebase fans. :-)
We are seeing this issue quite frequently in production (logged via sentry). We developed an ecommerce site for our client where we use firestore to track inventory and prices in real-time. The client went through some heat maps and found that prices weren't loading in some cases and via sentry we confirmed that a few visitors encounter "Could not reach firestore backend..." issue everyday.
After coming across this thread, I turned on the setting experimentalAutoDetectLongPolling
but it didn't make any difference. Can someone please suggest a workaround or a fix? It's very important that we fix this urgently to prevent our client from losing potential customers. I suppose calling firestore via cloud function will be more reliable since that will be over http. However that will require us to make several changes since we depend quite a bit on the real-time benefits of firestore.
@nikhilag - Have you tried using experimentalForceLongPolling
? It's a temporary workaround while we get auto detection working.
Thanks @rafikhan. I just deployed with this setting. Will update here again after a day to check if it worked.
@rafikhan It doesn't seem experimentalForceLongPolling
worked for me either. I still got similar no. of issues in Sentry today. It could be our site is experiencing some issue which is unrelated to this thread.
@nikhilag Can you tell us anything about the network connection your users are on that might be hitting this issue? How reliable is it? Is it 4G, etc.?
@rafikhan I don't have the data on the network connection but most people seem to be on 4g based on the heat maps. Everything in the site loads properly except for the prices from firestore. I saw a very high number of "Could not reach Cloud Firestore backend" issues today.
It seems to me that this is a hard issue to solve. I came across multiple issues online and none of them have a good solution:-
https://github.com/firebase/firebase-android-sdk/issues/1794 - July 21, 2020 https://github.com/firebase/firebase-android-sdk/issues/1790 - July 20, 2020 https://github.com/firebase/firebase-js-sdk/issues/2923 - April 16, 2020 https://github.com/firebase/firebase-js-sdk/issues/1457 - Jan 8, 2019 https://groups.google.com/g/firebase-talk/c/uLRHxgqtQaE?pli=1 - June 17, 2020
I suppose I have to start using cloud functions to pull data from firestore and I can just use polling to keep the data up to date. I think firestore realtime feature is awesome but I need to solve this issue for our client urgently and it seems rest api might be my best bet. One problem with cloud function is that the cold start takes too long. I can use cloud run with minimum one instance running to ensure better response time. If there's any alternative strategy I can try, please let me know. Thanks!
Thank you so much for publishing your log! We made some recent changes to the debugging tool at https://debug-my.firebaseapp.com/ . Can you please use it again and submit a new gist?
@scottfr @chenlevy @pimmee @Benny739 @catchshyam - It would be super helpful if you could do the same.
@rafikhan something I wanted to ask: I'm betting the debugging tool should be run (in my case) in the Emulator's browser, correct? Does it matter if experimentalForceLongPolling
is set in my app running on the same device?
@benomatis yes, in your case use the emulator's browser. The setting between each firestore instance is independent. It shouldn't interfere. If there's an issue we'll see it in the logs.
@rafikhan I updated my comment with the new logs. Thanks for your support!
The output looks like it was still from the old tool. I think the caching on the tool was not setup correctly. Can you clear the browser cache (or do hard reload) on your emulator and try again?
@rafikhan I ran it again after cache clear, edited my comment with the new gist link.
@benomatis the debug page shows a healthy connection to the outside world which suggests the issue is something specific within your app. Are you using libraries that hook into network calls like sentry.io, etc.?
@rafikhan No, not really, the stripe 3rd party package I use has not been implemented at the time I experienced the issue.
Here is my dependency list for reference, if that helps: https://gist.github.com/benomatis/2449edacd2a900b03eb8cefe1fac87f9
Hi @benomatis - Can you update your react native app to use experimentalAutoDetectLongPolling
and let us know what the observed behavior is? Thank you.
@rafikhan It completely resolves the the issue. Previously the app could not connect to firebase, with this setting on it now can. This is an Android-only issue, though I guess this might have been self-explanatory to some already.
Hi, I wanted to let you know about my observations related to this issue.
Generally default settings are ok for most users, but as you already know there are some specific networks or maybe other network environment settings that cause connection problems for some users. When we changed firestore settings to experimentalAutoDetectLongPolling it worked for the users, that had connection problems before, but... for many other users, that didn't have any issues with the connection we noticed that error "Could not reach Cloud Firestore backend" was thrown quite often. So we switched to experimentalForceLongPolling and all the issues was gone.
So it seems that experimentalAutoDetectLongPolling is working, but introduce another problem. We check that with firebase-js 8.4.2 and 8.6.7.
Hey, I've been having the issue described in the original comment when using firebase emulators (so, all network connections should be local).
Hopefully the information provided is useful?
What environment causes the problem (app platform, antivirus software, network proxy, network conditions, etc.)?
Firebase (npm module used in React.js): 8.6.8 Antivirus: Windows, Sophos. As far as I can tell, these don't detect anything suspicious Network: It's all on the same PC so network connection should be ideal. All connections to the actual firebase api works perfectly I've documented the experience on StackOverflow in the hopes others could help (I've only just found this issue after hours of searching).
What is the behavior without experimentalForceLongPolling or experimentalAutoDetectLongPolling?
Connections to a specific firestore endpoint never completes (except if I manually go to the URL in another tab).
URL: http://localhost:8081/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fmy-project%2Fdatabases%2F(default)&VER=8&RID=rpc&SID=FgqppSIdbuIaPmABDJD6fA%3D%3D&CI=0&AID=0&TYPE=xmlhttp&zx=yzaslurbjssn&t=1
Does experimentalAutoDetectLongPolling completely resolve the issue? Does experimentalForceLongPolling completely resolve the issue?
As far as I can tell, yes. Haven't completely tested it yet as its getting late but, a quick looks suggests adding these settings have fixed the issue.
Please visit https://debug-my.firebaseapp.com/, wait for "All tests done" (about 60 seconds), paste the results into a gist, and paste the link in your comment.
I don't think this is necessary as only the local emulators seems to cause this issue. In regards to that, if this is the wrong place to be posting this comment, I appologise.
Edit: It may be a Windows problem or, a configuration issue on my specific machine. I booted up a Linux VM and it seems that it works perfectly without the experimental function calls...
What environment causes the problem (app platform, antivirus software, network proxy, network conditions, etc.)?
I have an app built with Expo and it's been working fine. I exported the code ( expo run:android
) and immediately could not connect to Firestore from the app, both on real devices (Pixel 2) and an emulator (Pixel 4a running on 64-bit emulator, v 30.7.5-7491168 on a Mac). It is set up to use Android Studio HTTP proxy.
What is the behavior without experimentalForceLongPolling or experimentalAutoDetectLongPolling?
I get this error: "@firebase/firestore: Firestore (8.7.0): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds. This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend."
Does experimentalAutoDetectLongPolling completely resolve the issue?
It seemed to work sometimes after restarting the app. It's using react native so sometimes just editing the change and saving it would cause the JS to reload and then the app got past the error. Deleting the entire storage of the app and forcing it to go through the complete initialize would never succeed.
Does experimentalForceLongPolling completely resolve the issue?
Yes, although I've only been able to test a little thus far today. It definitely has been consistently successful thus far.
Please visit https://debug-my.firebaseapp.com/
I ran the requested URL, above, on from Chrome on the emulated device and no tests failed. https://gist.github.com/fitterman/d6d61adf2ad527cec5accbad0d0353d7
I did try it from the Mac (Firefox) that the emulator is on and one test failed. I am not sure this is relevant: https://gist.github.com/fitterman/6e3c7949f2875405e1035491dff0399f
Hi, I wanted to let you know about my observations related to this issue.
Generally default settings are ok for most users, but as you already know there are some specific networks or maybe other network environment settings that cause connection problems for some users. When we changed firestore settings to experimentalAutoDetectLongPolling it worked for the users, that had connection problems before, but... for many other users, that didn't have any issues with the connection we noticed that error "Could not reach Cloud Firestore backend" was thrown quite often. So we switched to experimentalForceLongPolling and all the issues was gone.
So it seems that experimentalAutoDetectLongPolling is working, but introduce another problem. We check that with firebase-js 8.4.2 and 8.6.7.
This is my experience with experimentalAutoDetectLongPolling aswell. And not hard to reproduce this error, just refresh the app a few times and it randomly happends.
We are having this issue, experimentalAutoDetectLongPolling
does not help but experimentalForceLongPolling
does. With limited testing we haven't noticed any performance impact with long polling enabled, can you shed some more light on what sort of performance issues we might see?
I'd like to find a way of implementing what I presume the auto-detection is trying to do (I can't find where in the source code it is implemented) so the polling is only switched on for clients that truly need it. We don't want to force users to use polling if they don't need to and I'd rather not have to ask clients to switch this option on themselves through using a checkbox or something.
Here is a gist from one of the affected machines. They are both running Windows 10 and using Chrome browser, I don't know the exact reason why they can't connect but they are corporate supplied machines so I assume there is some kind of firewall software blocking the connection.
I'd also like to know what kind of performance impact we can have with long polling enabled
Also had this problem, experimentalAutoDetectLongPolling
did not work, but experimentalForceLongPolling
fixed all problems. Unfortunately I'm not aware which software or configuration was blocking it.
I find this crazy that we can't even catch the error with a try-catch! Am I missing someting?
@baptisteArno
I'd also like to know what kind of performance impact we can have with long polling enabled
The client may see up to 1-RTT extra delay for individual server-sent messages. If the server-sent message rate is < 1/1-RTT, the latency impact is going to be negligible.
It might also be easier to understand if you take into account how long polling works. Once the client receives a message, the connection is dropped. The client then re-opens the connection and waits for the next event. If the next event already occurred by the time the client reconnected, there is extra latency. If the next event has not yet happened, the connection is upheld until the next event occurs, in which case there is no latency impact. The extra latency is therefore limited to the time of a roundtrip.
In our app when we set experimentalAutoDetectLongPolling: true
AND do NOT call firestore.enablePersistence(…)
we've been getting a lot of the errors below. It seems to only happen with this combination. Calling firestore.enablePersistence(…)
OR removing experimentalAutoDetectLongPolling: true
seems to resolve the issue. Is anyone else running into this?
Interestingly I do NOT seem to run into this issue when using experimentalForceLongPolling
instead of experimentalAutoDetectLongPolling
.
@squinn @MarkChrisLevy @tgangso @fabiank0 @btjones Thanks so much for all of your bug reports!! 😃
We're sorry for some earlier delays, but we're back on it now! :)
We're extremely interested in the case where experimentalAutoDetectLongPolling
doesn't work while experimentalForceLongPolling
works. And even more so, when experimentalAutoDetectLongPolling
causes regressions to users who were able to connect fine without the flag! (I know this one doesn't always consistently repro.)
It would be tremendously helpful if you could, from the problematic machine/browser, visit https://debug-my.firebaseapp.com/ and share the complete test output (it can take a few minutes for all 7 tests to finish) through a gist.
Thanks so much in advance! 😃
Id Like to leave my test results I received.
https://gist.github.com/srand2/6d4dcdb9832fffdc7abfbf1869211ff1
I was facing the timeout error while using burpsuite proxy and trying to reach examplesite.com
Any help would be appreciated
@sampajano here are my results: https://gist.github.com/btjones/a2a2661e858bdbdc6a06b3801f8859a8
@srand2 @btjones Thanks so much for sharing your the test results!!
@srand2 In your test result we can see both detectBufferingProxy
tests took over 30 seconds to finish.. Definitely demonstrating the issue! We'll look deeper into the logs!
@btjones In your test result we saw the detectBufferingProxy test with webchannel.googleapis.com
finish within 2-3 seconds but the one with Firestore (test no. 6 has a typo it should be detectBufferingProxy instead) took 30+ seconds. We'll look into the reasons why.
Thanks again! :)
@sampajano there was a log in my comment too, not sure if you missed it. https://github.com/firebase/firebase-js-sdk/issues/1674#issuecomment-884296844
@alexstanbury Yes! Thanks a lot for reminding me! I've seen it but I've misread the log when i looked earlier.. 😃
Your log is showing the same behavior as the report by @srand2 above! Thanks again for providing the report! We'll take a deeper look and have some diagnostics (and hopefully fixes) soon! 😃
@sampajano Hi, I am developing an application that @srand2 is trying to access.
Here is the part of the code that related to the issue:
That did not fix the issue @srand2 is facing.
Any help would be appreciated.
@DavidWeiss2 thanks for the report! We're able to reproduce this issue now and is working on root causing and fixing! Thanks for your patience in the meantime! 😃
Hi, I am having problems with the firestore while using the Emulator. I tried both experimental props in settings(), without success. Here are one warn and one error, that are raised when get()'ing a dummy doc:
I think I know where the lib error is. NOTE THAT there is a wrong url generated there in the object printed. Search for http://http://
here and you will find it. Looks like it's appending to the start another 'http://' for some reason. I use the same url for the Auth and it's working properly, so my url is correct ('http://192.168.0.18', port '8080').
Edit: Looks like that for useEmulator(host, port)
, I shouldn't pass the url in the host. Just passing '192.168.0.18' worked! The code should really check if the http:// is already included or complain if it's already included and it shouldn't be there. I lost a good hour on that :/
Are you experiencing the error
Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds
despite your network seeming healthy? Please visit https://debug-my.firebaseapp.com/ and check the results at the end. If the tests "with default options" fail (or are very slow) but the "with forceLongPolling" ones succeed, then this indicates your traffic is likely going through a proxy that is buffering responses in a way that is not compatible with Firestore.As a workaround, you can force long-polling as follows:
Have you enabled
experimentalForceLongPolling
andexperimentalAutoDetectLongPolling
to solve a reproducible connection issue related to a specific environment?We would like to know:
experimentalForceLongPolling
orexperimentalAutoDetectLongPolling
?experimentalAutoDetectLongPolling
completely resolve the issue?experimentalForceLongPolling
completely resolve the issue?If there is an existing comment concerning the environment you are using, feel free to just add a 👍 to it.