Add file.getSignedUrl() support in Storage Emulator

[jsakas]

Environment info

Running inside firebase functions (also emulated)

"firebase": "8.6.1",
"firebase-admin": "^9.8.0",
"firebase-functions": "^3.14.1",
"firebase-tools": "^9.11.0",

Platform:

macOS Big Sur 11.3.1 (20E241)

Steps to reproduce

Inside any cloud function which processes files and needs to sign a URL:

    const [file] = await bucket.upload(tmpFilePath, { destination: output });
    const signedUrl = await file.getSignedUrl({
      action: 'read',
      expires: addMinutes(new Date(), 60 * 24).toString(),
    });

Expected behavior

I receive a signed URL.

Actual behavior

I receive this error:

[emulators] >  {"verifications":{"app":"MISSING","auth":"MISSING"},"logging.googleapis.com/labels":{"firebase-log-type":"callable-request-verification"},"severity":"INFO","message":"Callable request verification passed"}
[emulators] >  {"severity":"ERROR","message":"Unhandled error Error: Cannot sign data without `client_email`.\n    at GoogleAuth.sign (/Users/jonsakas/Development/GuestHouse/guesthouse-cms/node_modules/google-auth-library/build/src/auth/googleauth.js:631:19)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)\n    at async sign (/Users/jonsakas/Development/GuestHouse/guesthouse-cms/node_modules/@google-cloud/storage/build/src/signer.js:97:35) {\n  name: 'SigningError'\n}"}

Alternatively, following the documentation that says a project ID is required in the initializeApp I receive a different error:

[emulators] >  {"verifications":{"app":"MISSING","auth":"MISSING"},"logging.googleapis.com/labels":{"firebase-log-type":"callable-request-verification"},"severity":"INFO","message":"Callable request verification passed"}
[emulators] >  {"severity":"ERROR","message":"Unhandled error FirebaseError: Bucket name not specified or invalid. Specify a valid bucket name via the storageBucket option when initializing the app, or specify the bucket name explicitly when calling the getBucket() method.\n    at new FirebaseError (/Users/jonsakas/Development/GuestHouse/guesthouse-cms/node_modules/firebase-admin/lib/utils/error.js:44:28)\n    at Storage.bucket (/Users/jonsakas/Development/GuestHouse/guesthouse-cms/node_modules/firebase-admin/lib/storage/storage.js:104:15)\n    at /Users/jonsakas/Development/GuestHouse/guesthouse-cms/functions/lib/http/downloadAll.js:54:35\n    at async func (/Users/jonsakas/Development/GuestHouse/guesthouse-cms/node_modules/firebase-functions/lib/providers/https.js:336:26) {\n  errorInfo: {\n    code: 'storage/invalid-argument',\n    message: 'Bucket name not specified or invalid. Specify a valid bucket name via the storageBucket option when initializing the app, or specify the bucket name explicitly when calling the getBucket() method.'\n  }\n}"}

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[abeisgoat]

We currently only have minimal API support for the Cloud API interface, which sadly is completely different and substantially larger than Firebase's, so we're not aiming for 100% compatibility because of this.

Signed URLs are not currently planned to be supported, but we can leave this open for a bit and if you're interested in signed URLs just hit this message with an emoji reaction and we'll keep an eye on it and prioritize accordingly.

[DibyodyutiMondal]

I was thinking... When using the emulator, we don't really need an actually signed url, just a url that works with the given request. For example, emulator auth jwt tokens don't have the signature after the payload, but it works.

So, in the storage emulator, perhaps the 'signed url' could simply be the object's url, plus a uuid token (similar to getDownloadUrl), with the difference that, this uuid token is not long-lived and adheres to the permissions and settings provided while calling getSignedUrl()

So, while in production, getSignedUrl() uses the gcloud storage library, but if it detects the emulator - it just uses a stubbed method instead.

Use case

Imagine something like online education - course videos are uploaded and only people who have bought a course are able to view them.

1. Using custom claims is the best way forward initially, but that limits the max number of courses a user can buy, because the size of the auth token is limited.
2. Access control lists are also a possible way, but how does that work at scale? I haven't been able to find much info about this, but I do not think we can have objects with huge ACLs including thousands (or millions) of users. In the absence of clear information in that regard, I'm doubtful of going down this route.
3. We could use node.js and app engine to stream the files to authenticated users. But that ends up destroying a lot of the built-in advantages of using firebase cloud storage. And it also costs more, I suppose - the app engine will have to be alive for the entirety of the stream and have to scale for the many streaming requests from many users.
4. The best way, is to use getSignedUrl(). First, we prevent all direct reads from firebase storage using storage rules. And in a cloud-function or app-engine, we could check firestore or wherever to see if our user has access to the course, and then make a signed url for the video being requested, if access is allowed, and send that off to the client so that they can stream it on their device. Thus, it's extremely scalable and secure. The only drawback? You will not be able to use storage emulator for this method, having to use production services instead. If there are other parts of your app which are more simple and straight while using storage, then using production cloud storage will also force you to use production auth service (emulator auth tokens won't work). And if you make a mistake, you can't "go back" to a previous state by restarting the emulator.

In my experience, the same hurdle also exists for something like an e-commerce app where I want content to be moderated before being published. And I'm sure there will be others. Using getSignedUrl simplifies a lot of things. It's not like the current emulator doesn't work - it just messes with the dev experience for advanced cases involving storage.

[weixifan]

I filed an internal issue b/197475725 to track this feature request. Please remember to add an emoji reaction to the post by @abeisgoat above if you are interested in this feature and we will prioritize accordingly.

[DibyodyutiMondal]

Any updates on this?

[nabid-pf]

Waiting...

[nicolls1]

Since it doesn't seem this will get attention soon, you can use the public url when the function is running in the emulator for now. Here's how to do that for a list of files:

      const [images] = await bucket.getFiles({
        prefix: `profile/${profile.id}`,
      })
      const urls = await Promise.all(
        images.map(async (image) =>
          process.env.FUNCTIONS_EMULATOR
            ? image.publicUrl()
            : (
                await image.getSignedUrl({
                  version: 'v4', // Allow to set long expire timestamps
                  action: 'read', // Read Only
                  expires: new Date().getTime() + 24 * 60000, // 24 hours from now
                })
              )[0]
        )
      )

[gOzaru]

@jsakas You are missing Authentication info. I think you should not use emulator for this case of use. You should process it directly inside Google Cloud Platform. It is such a waste of time to rely on emulator. I prefer using real-time updates from GCP itself, compared from this emulator.

[noahbrenner]

I think it's important to have some sort of implementation for this method in the emulators suite. I prefer not to develop/test in production. In my view, the reason for the emulators is to enable implementing and testing features before making any changes to production configuration and to use the same code paths in dev, test, and prod, to the extent possible.

[colohan]

First off -- thank you so much to @nicolls1 for posting a workaround! Very helpful. I wasn't sure how to detect whether I was executing in an emulator environment or not, and this makes it crystal clear.

Second -- if the Firebase emulator is not planning on supporting this in the short term, would it be possible to change the error message to be more helpful, so that users don't have to spend too much time puzzling over what is wrong (like I did)? It would be great if the error message was something simple like "getSignedUrl() not fully supported in the emulated environment, see https://github.com/firebase/firebase-tools/issues/3400"

[patrickdundas]

Following up on this. This is still a very much needed feature. Could we get a status update?

[dikatok]

@taeold is implementing publicUrl for write in storage emulator still open for contribution?

[dikatok]

I saw 2 open PRs regarding getSignedUrl support for emulator, are they abandoned? https://github.com/firebase/firebase-tools/pull/6142 https://github.com/firebase/firebase-tools/pull/6068

[dikatok]

by the way, a temporary solution that works for me is to create a local-only firebase function to receive the upload request and do the uploading, so something like this.

Inside API to generate the pre-signed URL

    const filename = randomUUID();
    const ref = firebase.storage().bucket().file(filename);
    const uri = ref.cloudStorageURI.href;
    if (configs.USE_EMULATOR) {
        return {
            uri,
            signedUrl:
                "http://192.168.1.4:5001/demo/asia-southeast1/upload/" +
                filename,
        };
    }
    const [signedUrl] = await ref.getSignedUrl({
        version: "v4",
        action: "write",
        expires: Date.now() + 60 * 10000,
        contentType: "application/octet-stream",
    });
    return { uri, signedUrl };

Firebase function to intercept the upload request

export const upload = functions.https.onRequest(
    { omit: configs.NODE_ENV !== "development" },
    async (req, res) => {
        const filename = req.params[0];
        const file = req.rawBody;
        await firebase.storage().bucket().file(filename).save(file);
        res.status(200).send({});
    },
);