"Cannot parse Firebase url" on snapshot reference access when using RTDB via firebase functions.

filipesilva commented 4 years ago

[REQUIRED] Describe your environment

Operating System version: Windows 10
Browser version: Chrome 84
Firebase SDK version: 7.16.1
Firebase Product: database

[REQUIRED] Describe the problem

It's currently not possible to access the data snapshot reference when using RTDB via firebase functions.

I believe this is due to the check in https://github.com/firebase/firebase-js-sdk/blob/6af4c27743372ba531e8ce3d046ae2f81e8f5be1/packages/database/src/core/util/libs/parser.ts#L83-L91

That piece of code checks that parsedUrl.domain is localhost, but it seems to be emulator-test-1.localhost instead.

Steps to reproduce:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

You should see the following error log in the console:

i  functions: Beginning execution of "postsPushHandler"
>  [2020-07-22T15:49:53.227Z]  @firebase/database: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
!  functions: Error: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
    at fatal (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:341:11)
    at parseRepoInfo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:1296:9)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:14990:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at DataSnapshot.get ref [as ref] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\providers\database.js:293:34)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:17:24
    at cloudFunction (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\cloud-functions.js:132:23)
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:573:20
!  Your function was killed because it raised an unhandled error.

Relevant Code:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
  console.log(snapshot.ref)
});

filipesilva commented 4 years ago

cc @samtstern (because I've reported other emulator related issues to you)

samtstern commented 4 years ago

@filipesilva what version of firebase-functions are you using? This should be fixed in version 3.8.0 (assuming newest version of the CLI as well)

filipesilva commented 4 years ago

The repro (https://github.com/filipesilva/firebase-emulator-parse-url) is using firebase-functions@3.8.0. Everything there is the latest as of today, I believe.

samtstern commented 4 years ago

@filipesilva sorry I should have clicked into that

samtstern commented 4 years ago

Ok a few things here:

I was able to reproduce this
If I change the initialization block to just admin.initializeApp() this error goes away.

samtstern commented 4 years ago

It looks like parseRepoInfo() is being called with a URL like:

https://emulator-test-1.localhost

This is a firebase-functions issue, transferring.

samtstern commented 4 years ago

Ok so I added some logging and extractInstanceAndPath is being called like this:

>  extractInstanceAndPath(projects/_/instances/emulator-test-1/refs/posts/-MCvh_klBc9LYP7JIlsJ, localhost)
>  return [https://emulator-test-1.localhost,/posts/-MCvh_klBc9LYP7JIlsJ]

The dataConstructor in onCreate is being given this raw.context:

{
  "eventType": "google.firebase.database.ref.create",
  "params": {
    "postId": "-MCviHJhohdsNE5uAJWZ"
  },
  "domain": "localhost",
  "resource": {
    "service": "firebaseio.com",
    "name": "projects/_/instances/emulator-test-1/refs/posts/-MCviHJhohdsNE5uAJWZ"
  },
  "timestamp": "2020-07-23T12:49:58.150Z",
  "eventId": "5/3OSuz1TP5pWvc/Lt0o9RDcN2M=",
  "authType": "ADMIN"
}

So the port is lost ... gonna have to think about this one.

samtstern commented 4 years ago

@filipesilva question for you: when you're running inside the Functions emulator and you also have the RTDB emulator running, but you do this:

admin.initializeApp({
   databaseURL: "https://emulator-test-1.firebaseio.com/",
   credential: admin.credential.applicationDefault()
});

const db = admin.database()

Do you still expect db to point to the emulator? To me it looks like you're explicitly trying to access production, otherwise you could do admin.initializeApp() to accept the defaults.

So I'm trying to decide if the actual bug is that your posts function writes to the emulators at all.

filipesilva commented 4 years ago

@samtstern I don't quite care about setting credential and would be happy enough if that is just defaulted.

I do care about setting databaseURL because I test a sharded RTDB and it is important that functions I have can be routed to the correct shard, and that the shard is using the emulators. I think this is what happens today because, in my particular work setup, when I save security rules, it will update rules for all 100 shards I use, and they show up in the emulator UI. I could use a URL that's pointing to the emulators by encoding that as a build time variable.

I also care about setting databaseAuthVariableOverride as well because I want my writes from firebase functions to still obey validation rules.

Attempting to set it with no databaseURL (e.g. admin.initializeApp({databaseAuthVariableOverride: "something"}); ) results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Can't determine Firebase Database URL.
    at FirebaseDatabaseError.FirebaseError [as constructor] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:43:28)       
    at new FirebaseDatabaseError (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:204:23)
    at DatabaseService.ensureUrl (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:89:15)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:56:26)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:8:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
    at Generator.next (<anonymous>)
!  Your function was killed because it raised an unhandled error.

Attempting to set it with databaseURL but no credential also results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Only objects are supported for option databaseAuthVariableOverride
    at new Repo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:12513:27)
    at RepoManager.createRepo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15049:16)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15014:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:10:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
!  Your function was killed because it raised an unhandled error.

So in my concrete work example it does not seem possible to just go with the defaults. I must set credential and databaseURL in order to set databaseAuthVariableOverride. In some cases I must also set databaseURL to access shards.

samtstern commented 4 years ago

@filipesilva thanks for clarifying all of that ... there's a lot going on here and I'll have to think of the best way to solve it.

filipesilva commented 3 years ago

@samtstern heya, did you have time to think about what a resolution for this looks like?

samtstern commented 3 years ago

@filipesilva thanks for bumping this issue! I took another look today and think I found a really simple fix: https://github.com/firebase/firebase-functions/pull/838

Here are the functions I am using:

const functions = require('firebase-functions');
const admin = require('firebase-admin');

const instance = "fir-dumpster-secondary";

admin.initializeApp({
  databaseURL: `https://${instance}.firebaseio.com/`,
  credential: admin.credential.applicationDefault()
});

exports.posts = functions.https.onRequest(async (request, response) => {
  const db = admin.database();
  const ref = await db.ref('posts').push({
    date: new Date().toISOString()
  });
  response.send(`Added: ${ref}`);
});

exports.postsPushHandler = functions.database
  .instance(instance)
  .ref('/posts/{postId}')
  .onCreate(snapshot => {
    console.log("onCreate:", snapshot.ref.toString());
    return true;
  });

And here are the logs I get:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  onCreate: http://localhost:9000/posts/-MPslO6MOO53iuSWFUJL
i  functions: Finished "postsPushHandler" in ~1s

In the Emulator UI it all seems wired up correctly: Screen Shot 2020-12-31 at 12 29 25 PM

Would you mind testing this for me? You can check out my branch of this repo and then run npm run build. Then in your functions repo run npm install --save /path/to/your/clone/firebase-functions which should let you use the local version.

filipesilva commented 3 years ago

@samtstern thanks for getting back to me so quickly! I tested your branch build on https://github.com/filipesilva/firebase-emulator-parse-url and can confirm I no longer get the URL error, and that the write correctly goes through. I think that fixes it!

filipesilva commented 3 years ago

@samtstern today I was integrating your changes into our codebase, and I found a follow up problem with the fix you submitted.

I've updated the original repository with the most recent firebase versions and code to repro this problem in https://github.com/filipesilva/firebase-emulator-parse-url/commit/979d5c65808b15a423ee29aff0c33748c3c9a9eb.

The repro steps are still the same:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

This time the output is:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  snapshot ref http://localhost:9000/posts/-MXYBOwrxE0SXK0MOAmN
>  admin app options {
>    databaseURL: 'https://emulator-test-1.firebaseio.com/',
>    credential: RefreshTokenCredential {
>      httpAgent: undefined,
>      implicit: true,
>      refreshToken: RefreshToken {
>        clientId: '563584335869-fgrhgmd47bqnekij5i8b5pr03ho849e6.apps.googleusercontent.com',
>        clientSecret: 'j9iVZfS8kkCEFUPaAeJV0sAi',
>        refreshToken: '1//0dffIHm3tl_W7CgYIARAAGA0SNwF-L9IrkirUzBA22L9-rvZHCDa4jcM47eAhdK2USE-8miFIGJkYRn39CoFzhFTErl82HejNLJU',
>        type: 'authorized_user'
>      },
>      httpClient: HttpClient { retry: [Object] }
>    }
>  }
>  [2021-04-05T18:00:42.083Z]  @firebase/database: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
⚠  functions: Error: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
    at fatal (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:341:11)
    at RepoManager.createRepo (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15218:13)
    at RepoManager.databaseFromApp (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15185:25)
    at initStandalone (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15462:45)
    at Object.initStandalone$1 [as initStandalone] (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15594:12)
    at DatabaseService.getDatabase (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/database/database-internal.js:80:23)
    at FirebaseApp.database (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/firebase-app.js:158:24)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/functions/index.js:19:40
    at cloudFunction (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-functions/lib/cloud-functions.js:134:23)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:592:16
⚠  Your function was killed because it raised an unhandled error.

This logging comes from the updated handler:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
   console.log("snapshot ref", snapshot.ref.toString())
   console.log("admin app options", adminApp.options)
   console.log("admin app db", adminApp.database())
 });

Attempting to access the database for the initialised admin app results in an error. There is no error if console.log("snapshot ref", snapshot.ref.toString()) is commented out.

It seems that using the snapshot ref initialises a database for the same url, but with a different URL.

Can this issue be reopened, or should I open a new issue?

samtstern commented 3 years ago

@filipesilva sure let's reopen this issue. So from that commit you sent it looks like there's a regression in one of the recent SDK updates. Could be firebase-admin, could be firebase-functions, etc. Do you think you could help me narrow it down by figuring out which of those changes in your commit leads to the regression?

filipesilva commented 3 years ago

Hi @samtstern! I tried going back to the original package versions for everything but firebase-functions in https://github.com/filipesilva/firebase-emulator-parse-url/commit/9ea2e64603b4f4d123fb248cce2cff012e964f95. The new problem still repros so I think it's related to firebase-functions.

I'm not even sure it's a regression though. I didn't test this originally with your fix because it required a larger time investment in updating our codebase, so it's possible that it was never working either.

firebase / firebase-functions