Closed anisabboud closed 7 months ago
I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
Ok, this is a lot of information but it seems like the core of it is that you get 403 attestation errors while the tab is in the background or the computer is asleep. I don't think the Firestore "terminate" thing is related.
403 attestation errors indicate that the ReCAPTCHA token sent to the AppCheck endpoint is bad, which could be for various reasons: it wasn't able to get any ReCAPTCHA token, or it got one and you got a bad ReCAPTCHA score (it thinks you are a bot or something), or the project API key doesn't match. The backoff for this type of error is set to 1 day. This is because it was assumed a ReCAPTCHA failure was a more or less long term problem that wouldn't somehow fix itself in a few seconds, unlike other errors such as a 503 (server too busy).
It might be possible to revisit this backoff policy and change it in the specific case of it being a 403 AND it being an attestation error (there's other types of 403s), to use a shorter exponential backoff, in case the ReCAPTCHA check failed due to the tab being in the background? (I'm guessing that's what happened.) This will be somewhat difficult because you can't tell on the client side what the ReCAPTCHA score was (as intended, to avoid exploitation).
Before I do that I want to do some more research into what happens to ReCAPTCHA when the tab is backgrounded, whether it's simply unable to reach the endpoint or whether it's actually giving you a failed score.
A bit more info that might help: I left two tabs open yesterday. This time they got the 403 after ~23 & ~24 hours (while in the background but the laptop was not asleep).
I filtered the network tab for firebaseappcheck
, and noticed that there were no "successful" POST requests to firebaseappcheck
.
I.e., the 403 is the first and only request to firebaseappcheck
. How often are POST requests typically sent to firebaseappcheck
?
Update: Retrying the experiment with two fresh tabs the next day.
This time woke the laptop from sleep after more than 24 hours (token expires after 24 hours).
Both tabs sent two (not one) POST requests to https://content-firebaseappcheck.googleapis.com/v1/...
, 10 seconds apart.
Requests throttled due to 403 error.
warning since the second request failed.)But since the first request succeeded on both tabs, this time both of them survived.
As mentioned above, there is a 1 day backoff when the response is a 403. I will try to look into some kind of solution for your situation that lets us maintain a long backoff for the cases it was intended for.
I'm still encountering this issue with Firebase 9.14. For instance today I encountered it while the computer was on and active, but the crashing tab was briefly in the background.
I noticed that tokens are sent with a TTL of 3600 (1 hour), so the token is refreshed every 55 minutes (as can be seen from the AuthService timestamps prior to the crash below):
The token refresh every 55 minutes is usually accompanied/followed immedately by a Firestore POST request with "Type: terminate" (I'm guessing to refresh the Firestore connection with the new token). Normally that flow works fine. Today for instance, the token was refreshed at 10:30, 11:25 and 12:20 (55 min apart), and each token refresh was followed by a terminate request to Firestore).
However, only 2 minutes later, at 12:22, the idle tab sent another terminate request to Firestore.
Since this request wasn't preceded by a refresh token, it might have triggered the 403 error.
This request was followed by two requests to content-firebaseappcheck.googleapis.com
(which is different from the token requests every 55 minutes that are sent to securetoken.googleapis
).
These are the only 3 network requests at the time of the crash:
I still don't understand what triggers this sequence of events...
I'm getting the same issue but with out leaving the tab open, I just get the error right away when trying to interact with firebase services. I'm setting up a contact form for my app and am hoping to use app check to block potential bots from submitting the form. I've set up app check with my other firebase services.
But when trying to add a document to firestore, it gets blocked due to app check (which is supposed to happen) but with the app check 403 error (which I have no clue why is showing up).
Another similar case today where the 403 was preceded by 504s:
same error here! anyone found a solution??
Just to be clear, a 403 error is not a bug in itself. It means that App Check failed attestation. That should happen when the app is not allowed to use App Check. That could mean reCAPTCHA failed or the app making the request is providing the wrong app ID, or there's a typo in your reCAPTCHA site key. This is what should happen in these cases. Further requests to the App Check backend are then throttled for 24 hours, and the reasoning behind this is that if any of these things are the case, it won't be fixed by retrying (whereas a 50x error might be), so it's throttled to not overwhelm your quota.
If you're getting this when you expect access to be blocked, that's normal. If you're testing things out in development you can clear your local indexedDB cache to get it to retry again without waiting 24h. If you're expecting access not to be blocked, make sure your reCAPTCHA account is set up properly and registered in the Firebase console.
If you feel like a 24h throttle is not appropriate in some cases (which seems like it might be the case), let us know the details, we are looking into what we can do to make it more granular somehow.
Hi, everyone.
I had the same problem, but after carefully checking it out based on @hsubox76 advice, I was able to solve it. ✅ I'll share the information for your reference.
First (and I don't know if this is truly relevant), I decided to generate a non-test Enterprise Key and use that. (Previously, I used a key with the Test Key flag enabled).
The code is as follows:
import { initializeApp } from 'firebase/app'
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check'
const firebaseConfig = {
// hide
}
const app = initializeApp(firebaseConfig)
initializeAppCheck(app, {
provider: new ReCaptchaEnterpriseProvider(import.meta.env.VITE_APP_RECAPTCHA_SITE_KEY),
isTokenAutoRefreshEnabled: true,
})
Next, I reconfigured the reCAPTCHA Enterprise
> reCAPTCHA Enterprise Site Key
for the target application as listed in Firebase > App Check > Apps
.
Finally, just to be safe, I disabled Firebase > App Check > Cloud Firestore
once and re-enabled it.
That's all there is to it. Incidentally, there was no need to wait 24 hours after this.
Best wishes for success.
I also started getting this error today, even though it's been working fine for the past year. Did not change any configuration.
I got the solution. The problem it's not you. It's the system. AppCheck to work without that error needs that the computer IP Address is the same when you register to AppCheck. My Router changed address my PC and I got a lot of errors that's the only one that remains until Changed the IP Address Back. I hope that helps. 😊
@hsubox76
403 attestation errors indicate that the ReCAPTCHA token sent to the AppCheck endpoint is bad, which could be for various reasons: it wasn't able to get any ReCAPTCHA token, or it got one and you got a bad ReCAPTCHA score (it thinks you are a bot or something), or the project API key doesn't match.
I am not getting this error when using normal browsing (Chrome), but when trying on incognito I get the error. A coworker of mine reported this issue to me today when he tried accessing our staging site and he gets this regardless of normal or incognito. We we're both able to access the site fine just yesterday.
The only assumption I can think of based on what you said is recaptcha thinks that I'm a "bad" user when using incognito. Is there a way to verify this? And are there any ways of telling appcheck to allow the user anyways, or perhaps lower the score threshold? Or allow additional verification should it be needed. From reading the recaptcha v3 features, this seems to be supported.
I made an error deploying an app with a wrong recaptcha key. OK , I understand it but now I have to tell my manager I can't work for 1d to fix this?
Hi guys, I just want to clarify that this issue is not about configuring AppCheck or Recaptcha for the first time. But rather it's specifically about an issue where a fully working app with AppCheck, throws a 403 error when the tab is left open/idle for a day or a few days. I.e., the app stops working randomly after being open for a few hours or days.
If you have a different issue, please open a separate issue to keep the discussion here focused on the main issue.
App Check debug token: abcdefgh-abcd-abcd-abcd-abcdefghijkl. You will need to add it to your app's App Check settings in the Firebase console for it to work.
Go to https://console.firebase.google.com/u/0/project/_/appcheck/apps → Apps tab → three dots ⋮ → Manage debug tokens → Add debug token → Paste the token from the Chrome console.
(In incognito, cookies/localStorage are cleared on every session and the debug token changes. So every time you open a new incognito session you need to add the new debug token again to the Firebase console, or you'll get a 403 error.)@hsubox76
403 attestation errors indicate that the ReCAPTCHA token sent to the AppCheck endpoint is bad, which could be for various reasons: it wasn't able to get any ReCAPTCHA token, or it got one and you got a bad ReCAPTCHA score (it thinks you are a bot or something), or the project API key doesn't match.
I am not getting this error when using normal browsing (Chrome), but when trying on incognito I get the error. A coworker of mine reported this issue to me today when he tried accessing our staging site and he gets this regardless of normal or incognito. We we're both able to access the site fine just yesterday.
The only assumption I can think of based on what you said is recaptcha thinks that I'm a "bad" user when using incognito. Is there a way to verify this? And are there any ways of telling appcheck to allow the user anyways, or perhaps lower the score threshold? Or allow additional verification should it be needed. From reading the recaptcha v3 features, this seems to be supported.
This is an interesting problem and worth looking into, but as mentioned above, can you open a new issue for this?
I think the focus of this issue should probably be cases where the currently configured 1d backoff (will not retry) after hitting a 403 error is inappropriate.
Cases it was intended for:
Retries in those cases will just blow out your quota for no good reason.
Cases where the 1d backoff might be inappropriate:
The last case I think is the initial problem brought up in this issue. The problem is that we have to figure out what state is causing the failure (offline? background tab? sleep?) and if there is a way to detect the browser being in that state, or detect coming out of that state. That part is going to be pretty tricky.
If anyone has any of the 2 cases above ("Cases where the 1d backoff might be inappropriate") and has any more insight into them, or can think of an additional case, please add in this issue. If it's not related to this, please open a separate issue.
@hsubox76
If you're testing things out in development you can clear your local indexedDB cache to get it to retry again without waiting 24h.
Thanks for the solution
Hey @hsubox76,
Is there any update on this at all? We're seeing this issue a lot in our logs and it's causing a lot of error noise. A lot of the time it's happening after the user comes back from being idle as previously mentioned in this thread.
It's hard to know exactly what experience the user is getting when this happens. Would love to know if there's a fix in the works or a temporary workaround? We're even starting to consider moving away from AppCheck as we're concerned it may be impacting conversions if we have no visibility on how it effects users.
Working on a PR that may address the problem if the issue is that reCAPTCHA is failing and not being caught, and the App Check endpoint is then being called unnecessarily: https://github.com/firebase/firebase-js-sdk/pull/7203
There still may be a lot of errors but they will say "recaptcha error" and a request won't be sent to the App Check exchange endpoint, which means you won't get a 403, which means further requests won't be throttled for 1 day. That means if the reCAPTCHA situation corrects itself (computer wakes up? tab is foregrounded?) the SDK will be free to send a new request to the App Check exchange endpoint that should work.
It seems like there may be multiple things going on in this situation so I'm not going to tag that PR as "fixes" this issue (which would close it) because there may be more to address.
Yep that's the issue @hsubox76 :)
Agree there's a few other issues going on here but that's the main one for us. Looking forward to the release and happy to test on this end when something is ready. I've subscribed to the PR so will keep track there. Thanks for the quick response.
Release 9.20.0 went out yesterday and should have the change described above.
This problem is still happening for us, and we've been on 9.20.0 for a few days now.
Today I experience the problem reliably when testing from my mobile phone when connected to the wifi. However, when I disconnected from wifi and used cellular data, immediately it started to work totally fine.
Something very strange going on here.
What errors, if any, are you seeing in the console? Do you see any ReCAPTCHA errors?
From LogRocket from the incident I described (after switching to cellular it worked), here's what I see:
@firebase/app-check:AppCheck: Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s (appCheck/throttled).
In another LogRocket where the problem also occurred, I see this. Not sure if's helpful to you:
@firebase/app-check: Failed to read token from IndexedDB. Error: FirebaseError: AppCheck: Error thrown when opening storage. Original error: A mutation operation was attempted on a database that did not allow mutations.. (appCheck/storage-open).
That sounds like it might possibly be a bad ReCAPTCHA result (ReCAPTCHA thinks you're a bot, when you log in from your phone on wifi). I noticed you didn't mention what exact problem you were originally seeing. Is it identical to that of the original post in this issue? If not can you describe what you're seeing?
See my earlier comment: https://github.com/firebase/firebase-js-sdk/issues/6708#issuecomment-1412549757 . Is it a sleep/background issue as described in the last bullet point there?
I see the exact same error regarding "Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s"
However, it is not related to sleep / wake.
When the problem occurs it happens immediately on first page load. Also clearing browser cache, and reloading browser does not fix the issue.
Can you file a separate issue and provide snippets or a repro of how you're using the code? This issue is centered around the sleep/background issue.
I experience the exact same issue. I didn't do anything. I just first load the page and maybe the recapcha failed at the first place. Then it throttle me and say I only be able to retry in one day...
@hsubox76
If you're testing things out in development you can clear your local indexedDB cache to get it to retry again without waiting 24h.
Thanks for the solution
I tried to delete the indexedDB but still get 403...
seems if one user get some issue, all the whole app check backend will give throttled for one day? I cleared the indexdb, changed the ip and browser(in fact changed a computer), but still my app check is throttled and get 403. Also, i USED a recaptcha enterprise test key, which should always good (always score 1)....
@bxclib2 Did you find a solution? This is an ongoing issue for our team for some time now. It is a very random problem. Sometimes it will work totally fine. Then, it will fail. One very odd thing I noticed is in Chrome, in normal browser, often it will work, but then on the same PC, if I open the same URL on a Private Chrome tab, then I will see the appcheck/throttled error!
It is very strange and inconsistent.
seems if one user get some issue, all the whole app check backend will give throttled for one day? I cleared the indexdb, changed the ip and browser(in fact changed a computer), but still my app check is throttled and get 403. Also, i USED a recaptcha enterprise test key, which should always good (always score 1)....
No, the throttle only applies locally, that data is stored locally. If you're getting 403s in new environments it's a fresh ReCAPTCHA failure, or at least a fresh App Check endpoint rejection. I don't know why your test key failed, does a test key still have to match the secret key on the back end? Could there be a mismatch there?
I am also facing the same error:
{ "error": { "code": 403, "message": "App attestation failed.", "status": "PERMISSION_DENIED" } }
the config is for deployed site and with enterprise key.
And the default ttl for the key is 30 minutes.
I'm copying and pasting a reply from another issue regarding what a 403 app attestation error means and how the App Check web SDK works in general. Please look through it before adding any additional comments here:
Some background on how App Check works on web - I'll try to be verbose here so I can reuse this explanation for other App Check issues.
The default attestation provider used is ReCAPTCHA (V3 or Enterprise) which I assume you are using. The first thing that happens when Firebase tries to get an App Check token is it uses ReCAPTCHA (with the site key you provided) to get a ReCAPTCHA token. If your environment fails the ReCAPTCHA check for any reason, you still get a token, but it's encoded that it failed. It's not designed to be decodable on the client side for obvious reasons (ReCAPTCHA is supposed to catch bad actors and you don't want them to be able to trial-and-error it).
The next thing that happens is that the Firebase SDK sends this ReCAPTCHA token (good or bad) to the App Check endpoint. Since you've given us your ReCAPTCHA secret key in the Firebase console, the App Check endpoint is able to use that to decode the ReCAPTCHA token and see if it's valid. If it's invalid, it's a 403 error.
At this point, it was decided to throttle any further requests from this client for 24h because it's unlikely that a bad ReCAPTCHA token will resolve itself on an immediate retry, and we didn't want people using up their quota because they keep sending the same bad ReCAPTCHA token to the App Check endpoint. (Other errors, like in the 500 range, where the server was busy or had some error, seem likely to be resolved with a retry in the next few seconds/minutes, so those have a short exponential backoff.)
So there's only two ways I know of to get a bad ReCAPTCHA token (and therefore the 403 which leads to throttling):
If the failure rate from ReCAPTCHA fails is unacceptable, one solution is to use CustomProvider
to write your own attestation provider, instead of using ReCAPTCHA.
Hey @anisabboud. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.
If you have more information that will help us get to the bottom of this, just add a comment!
Hi @google-oss-bot, the issue is happening less frequently after @hsubox76's improvements - thanks a lot!
But I've encountered it at least once after all the updates (i.e., AppCheck errors after returning to an idle tab, leading to Firestore disconnecting and the app stops working). I'll share more details the next time I observe this.
We are also still experiencing this issue, on v9.22.2
Update: This issue is occurring less frequently (thanks to @hsubox76's improvements), but unfortunately it still occurs.
Basically:
@firebase/app-check: AppCheck: Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s (appCheck/throttled).
FirebaseError: Missing or insufficient permissions.
=> App crashes.There are two types of token requests:
"securetoken"
requests for short."firebaseappcheck"
requests for short.securetoken
requests are sent every 55 minutes, whereas firebaseappcheck
requests are sent every 24 hours.securetoken
request is sent 55 minutes after initial page load, whereas the first firebaseappcheck
request is sent out after some seemingly random number of hours (could be ~1.5 hours, could be ~5 hours, etc.)firebaseappcheck
seem to fail with 403 when there is no "recently-successful" call to securetoken
.firebaseappcheck
(once every 24 hours), there has been a "recently-successful" request to securetoken
sometime in the past hour (since these are sent out every 55 minutes), so everything works fine.firebaseappcheck
is scheduled to happen, then when the laptop wakes up, an immediate request will be made to firebaseappcheck
and to securetoken
.
In this case, the request to firebaseappcheck
often precedes the request to securetoken
(by ~1 second), leading to a 403 (since there hasn't been any successful recent calls to securetoken
, leading AppCheck to think the user isn't legit)./securetoken|firebaseappcheck/
firebaseappcheck
will be made and will fail with 403.Assuming my analysis is correct (that the success of firebaseappcheck
requests relies on a successful previous recent securetoken
request), then before making a firebaseappcheck
request, there should be a check for the most recent request to securetoken
, and if there hasn't been any in the past hour, then wait until the one that has just been sent succeeds.
Great reproduction write up @anisabboud I was hoping it might mean a bit of action on this issue but its been 3 weeks. We have had to disable app check as it makes our site unusable for a significant portion of our users. Now investigating alternatives.
Assuming firebaseappcheck is failing because the recaptcha token has expired. Do you still have access to the response from the firebaseappcheck request? Im wondering if there is an error code or something that this js sdk could use to know it has to get a new recaptcha token.
Also from what i can see using the getToken(forceRefresh = true) does not force refresh the recaptcha token it only does the firebaseappcheck exchange using the same busted recapcha token.
Please help on firebaseappcheck alternative, as it was working for almost all users , but now only 30% users are able to use. Updated to latest version, but issue still persists.
I'm commenting here since the issue I opened https://github.com/firebase/firebase-js-sdk/issues/7116 was closed and locked. This error happened less often but still occurs.
Why is this error?
Please open a new issue if it is not related to failures after waking from sleep/background.
Edit: Also, for more info on a general ReCAPTCHA error, please see this comment above: https://github.com/firebase/firebase-js-sdk/issues/6708#issuecomment-1641069585
Thanks for the additional info @anisabboud but the "securetoken" endpoint is an auth endpoint and not related to App Check requests. I think from previous posts you've made, you may simultaneously be having two different issues upon sleep/wake, one with App Check, and one with Auth, which can get mixed up in the logs.
Regarding this issue, the lack of further solutions is not because it's not important to us, it's because we are unable to reproduce it. All of our previous fixes have been just guessing what might be going wrong on your machines and taking our best guess at addressing that.
One thing that might help us reproduce it faster is if it doesn't take 2 24 hour periods to attempt one reproduction. I would bet 24 hours is not some kind of hardcoded timeline but the TTL set in the console. Can you change it to 30 minutes and reproduce the same behavior over 2 30 minute periods instead? That seems like it would give us a better chance at reproduction.
Here's where you would change the App Check TTL in the console:
Additionally, if anyone has any other ways to quickly reproduce the error, especially if it's guaranteed to always get the same result, that's what would be most helpful in solving this issue.
Hey @anisabboud. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.
If you have more information that will help us get to the bottom of this, just add a comment!
Since there haven't been any recent updates here, I am going to close this issue.
@anisabboud if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.
I am experiencing a similar issue with Firebase AppCheck. When a tab with my website remains inactive in the background, returning to it often results in the 403 error.
I suspect the root cause might be linked to reCAPTCHA Enterprise's inability to gather sufficient user interaction data in inactive tabs, leading to failed human verification checks (false positives).
A potential solution could be allowing the option to disable or limit attestation request throttling in the SDK. This way, if an attestation fails, a new attempt could be made promptly, potentially resolving the issue. This could mitigate issues arising from false positives.
For context, I recently integrated Firebase AppCheck into my staging environment, a production-like setup (using a live Firebase project on a deployed website) restricted to internal testing. The occurrence of this issue in a live environment would be highly detrimental to user experience.
Is there any update from the Firebase team regarding a solution to this problem? @hsubox76
I've successfully replicated the problem across different devices and networks (different IP addresses). Here's a description of the steps I followed to reproduce the error:
Additional scenarios where the error occurs:
Context for my testing environment:
10.6.0
.
Summary
Using latest Firebase v9.12 (through latest AngularFire 7.4.1) in production, with AppCheck, Firestore, and Auth.
When leaving the webapp tab open and idle, the app often runs into a
403
error in POST request to exchange AppCheck token ("App attestation failed." "PERMISSION DENIED"), followed by a warning:@firebase/app-check: AppCheck: Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s (appCheck/throttled).
Console Tab
As you can see, the
appCheck/throttled
warning shows up twice, and is preceded by a failing 403 POST request tocontent-firebaseappcheck.googleapis.com/...exchangeRecaptchaV3Token
There was no exponential backoff. Quoting from @hsubox76 in https://github.com/firebase/firebase-js-sdk/issues/6373#issuecomment-1282744902:
After the failed token exchange & AppCheck throttling, Firestore can no longer load data, thus logging several "
FirebaseError: Missing or insufficient permissions.
" on all active document/collection listeners, rendering the app dysfunctional.In a second tab running the same app, a similar crash was observed, but it was preceded by a
Timeout
error inrecaptcha__en.js
and a@firebase/app-check: FirebaseError: AppCheck: ReCAPTCHA error. (appCheck/recaptcha-error).
Network Tab
As you can see from the network tab, the 403 POST request failed with "App attestation failed." - when is this supposed to occur?
There were no further requests to firebaseappcheck.googleapis.com, but there was a successful
OPTIONS
request200
at the same secondWed, 19 Oct 2022 10:17:35 GMT
:Right before the failing 403 request, during the same second 10:17:35 GMT, there was a POST request to Firestore with query parameter
TYPE=terminate
- could this be related? Sounds suspicious.There were no further requests to firebaseappcheck.googleapis.com but there were successful token requests to
securetoken.googleapis.com/v1/token
every 55 minutes before & after the 403 issue: 09:07:52 GMT 10:02:53 GMT 10:57:54 GMT 11:52:55 GMT 12:47:56 GMTReproduction
The issue is tricky to reproduce, as it's related to token expiration. I often leave the tab open for several days before it happens.
Today I encountered this issue during the workday, while the computer was on all the time, in two different tabs that were open in the background. The same behavior mentioned above was observed in both tabs, 4 seconds apart: Wed, 19 Oct 2022 10:17:35 GMT Wed, 19 Oct 2022 10:17:39 GMT
These 4 seconds could be the difference from when I initially opened or interacted with each tab, hinting at a token expiration issue timeout, possibly initiated by Firestore terminate.