Firebase appcheck token verification fails because the sdk can't retrieve JWKS

[hannutho]

Step 2: Describe your environment

• Operating System version: macos ventura 13.4
• Firebase SDK version: 11.10.1
• Firebase Product: appcheck
• Node.js version: 16
• NPM version: 8.18

Step 3: Describe the problem

When the sdk trys to fetch the jwks from google https://firebaseappcheck.googleapis.com/v1/jwks to verify the appcheck token it errors because the endpoint is returning 403 forbidden. It seems that google has added this rate limit for non authenticated identities lately. It can only be bypassed by passing a X-goog-api-key. In the sdk code for appcheck there is no api key passed nor is there the ability to pass an api key as an enduser.

ERROR:

Error: Error fetching Json Web Keys: Forbidden
    at AppCheckTokenVerifier.mapJwtErrorToAppCheckError (/opt/nodejs/node_modules/firebase-admin/lib/app-check/token-verifier.js:144:16)
    at /opt/nodejs/node_modules/firebase-admin/lib/app-check/token-verifier.js:119:24
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Runtime.handler (webpack:///./src/functions/authorization/appcheck/index.ts?:39:13)

I double checked and ran a couple of requests in postman against the jwks endpoint and received a forbidden error as well after some time.

Steps to reproduce:

1. setup firebase admin sdk
2. try to verify multiple appcheck tokens

Relevant Code:

const appCheckVerification = async (req, res, next) => {
    const appCheckToken = req.header("X-Firebase-AppCheck");

    if (!appCheckToken) {
        res.status(401);
        return next("Unauthorized");
    }

    try {
        const appCheckClaims = await getAppCheck().verifyToken(appCheckToken); !!!FAILS HERE!!!

        // If verifyToken() succeeds, continue with the next middleware
        // function in the stack.
        return next();
    } catch (err) {
        res.status(401);
        return next("Unauthorized");
    }
}

Using the sdk to verify an appcheck token does not work after hitting the rate limit of the jwks endpoint. It seems like an api key header needs to be passed in this line: https://github.com/firebase/firebase-admin-node/blob/a4019e491c9d36167853eaba056adebd0e9dfc87/src/utils/jwt.ts#L61

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[kapilvats]

I am also facing the same issue in our production environment,

GET https://firebaseappcheck.googleapis.com/v1/jwks

{ "error": { "code": 403, "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.", "status": "PERMISSION_DENIED" } }

[chocolatkey]

This was a disastrous change for code I had running in production. This change breaks the official SDKs' initialization, and prevented appcheck from working at all. I had to patch the SDK to include an API key in the URL in order to get my server running again, e.g. https://firebaseappcheck.googleapis.com/v1/jwks?key=<firebase web API key> but this is really an unacceptable breaking change.

Please either document this change and update the SDKs, or revert it

[dmelo]

I'm using the firebase-admin-python, but I have the same issue. On my local machine I'm able to retrieve the jwks, but on a AWS server, it returns a 403.

[mergoc]

we're experiencing the same problem

[rafaelcardoso]

same here 😢

[thuctd]

My error: FirebaseAppCheckError: Error fetching Json Web Keys: Forbidden at AppCheckTokenVerifier.mapJwtErrorToAppCheckError (node_modules\firebase-admin\lib\app-check\token-verifier.js:144:16) at \node_modules\firebase-admin\lib\app-check\token-verifier.js:119:24 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { errorInfo: { code: 'app-check/invalid-argument', message: 'Error fetching Json Web Keys: Forbidden' }, codePrefix: 'app-check' }

Node.js v19.1.0

Please help!

[yaozhiwang]

same here

[rarbo]

Yep. Same issue here too.

[SmartByt3r]

Same issue too Error: Error fetching Json Web Keys: Forbidden

[edisonpaul4]

We have the same problem, our production environment went down.

thanks firebase

[edisonpaul4]

WE need an apology.

[KarlaCarvajal]

Hi google team, we are having same problem, please help us this is affecting our final users on the production environment.

{"level":"error","timestamp":"2023-09-27T03:43:53.362Z","pid":228193,"hostname":"nicolas-Z590-VISION-G","message":"App check token validation failure","trace":"Error: Error fetching Json Web Keys: Forbidden","context":"Firebase Service"}

[MalcolmMcFly]

Same problem on our side. Has anyone discovered a workaround?

[gilesbutler]

Same here. This is unacceptable and we're looking at moving away from AppCheck now because of this. We lost hundreds of customers and sales yesterday. We were also up late and stressed trying to figure out exactly what was going on.

I've spent today implementing Cloudflare's Turnstile as a replacement and it gives you a lot more control and visibility to your captcha setup.

[chocolatkey]

@MalcolmMcFly check my comment above. You have to patch the SDK and add ?key=<firebase web API key> to the end of the hardcoded jwks URL.

[kapilvats]

Same problem on our side. Has anyone discovered a workaround?

We have temporary solved it with patching the firebase-admin node modules inside docker file

RUN sed -i 's#const JWKS_URL = '\''https://firebaseappcheck.googleapis.com/v1/jwks'\''#const JWKS_URL = '\''https://firebaseappcheck.googleapis.com/v1/jwks?key=$apiKey'\''#g' node_modules/firebase-admin/lib/app-check/token-verifier.js

[ctomichael]

Yep, this has broken production for us too. Come on Google, do better...

[aadito123]

Same here. Literally broke production with no warning.

[griseau]

It happened to us on europe-central2 deployment. It keeps working on europe-west, please don't release this change further 🙏

[F170x]

Same here

[Veigaum]

unbelievable

[srodriguez20]

Same here, production is broken

[coawazie]

Thought I was losing my mind! This is just wrong!

[weixifan]

Thank you everyone for swiftly identifying and reporting this issue. We have identified the issue and we are rolling out a fix now; once the fix is rolled out this issue should be resolved. This error affected a small portion of our users as we have gradual rollouts in place.

We really understand the frustration and we apologize for the inconvenience caused. We have added additional testing for the JWKS endpoint to prevent future recurrence.

[coawazie]

Just started working. Thank y'all for the fix.

[weixifan]

Thank you for your patience everyone. We have completed the rollout of the fix. Please let us know if you continue to experience issues.

We apologize again for the frustration this has caused.