Getting HTTP 400 error when trying to enable the Firebase extension through CLI

[jon-flowers]

Description

I am using the Firebase CLI to try to enable the extension for a Firebase project. Major note is that the last few times I did this it worked fine, so this seems to be a new issue. I think it may be related to this change: https://github.com/typesense/firestore-typesense-search/commit/44c5b37ce84ea2573cc40419f0e414865bbc4544

The TYPESENSE_API_KEY parameter appears to be the issue. Previously I had this set as the admin API key, which used to work. Now, when I try to enable the extension, I get an error related to secrets manager, which seems to be saying I have formatted it wrong. Specifically it says it should be formatted like projects/*/secrets/*/versions/* instead of just the API key, but I am not sure exactly how to format my API key like that (I am not sure which each of the asterisks in this case should be)

Steps to reproduce

Set up parameters file Run Firebase command to enable extension

firebase ext:install typesense/firestore-typesense-search --project=projectId --params=params.env --force

Expected Behavior

The extension will enabled for my Firebase project

Actual Behavior

I receive a HTTP 400 error:

Error: HTTP Error: 400, Secret ID 123.... is invalid, must be of form projects/*/secrets/*/versions/*

Metadata

Typesense Version: v0.23.0.rc27

OS: MacOS

[jasonbosco]

@jon-flowers Could you try installing via the Firebase web console? It did ask me to provision a new secret during installation and I just had to paste the Typesense API key and click on "Create Secret" in the UI.

@dackers86 Any thoughts on how to do this via the CLI?

[jon-flowers]

Thanks @jasonbosco , I checked and can confirm that I can use the UI to activate the extension. The flow has definitely changed slightly, with the API key now asking me to create a secret, but if I click that it seems to create fine.

That being said, based on the number of environments we have, we definitely would prefer to do this with the CLI / programmatically wherever possible

[jon-flowers]

Just checking in to see if there were any updates here, I'm blocked from rolling out Typesense to more environments unless I can automate the extension installation. Thank you!

[jasonbosco]

@jon-flowers I just tried this via the CLI and it seemed to work for me:

$ firebase ext:install typesense/firestore-typesense-search --project=typesense-extension-testground
i  extensions: ensuring required API firebaseextensions.googleapis.com is enabled...
✔  extensions: required API firebaseextensions.googleapis.com is enabled
i  extensions: Checking project IAM policy...
✔  extensions: Project IAM policy OK
i  extensions: information about 'typesense/firestore-typesense-search@latest':
Name: Search Firestore with Typesense
Publisher: typesense
Description: Indexes data from Firestore into Typesense for full-text search
License: Apache-2.0
Source code: https://github.com/typesense/firebase-typesense-search-extension/tree/master

i  extensions: This extension is in preview and is built by a developer in the Extensions Publisher Early Access Program. Its functionality might change in backward-incompatible ways. Since this extension isn't built by Firebase, reach out to typesense with questions about this extension.

It is provided “AS IS”, without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the extension, and its functionality might change in backward - incompatible ways.

i  Want to review the source code that will be installed? Download it here: https://storage.googleapis.com/firebase-mod-sources-prod/a8ccc68575ba7005977665b9bdb68e0e012aadceb2579ef589f8f154d1181bb9

? Do you wish to continue? Yes
i  extensions: Search Firestore with Typesense will enable the following APIs for project typesense-extension-testground
- secretmanager.googleapis.com: To access and manage secrets which are used by this extension. By using this product you agree to the terms and conditions of the following license: https://console.cloud.google.com/tos?id=cloud&project=typesense-extension-testground
? Do you wish to continue? Yes
i  extensions: ensuring required API secretmanager.googleapis.com is enabled...
✔  extensions: required API secretmanager.googleapis.com is enabled
i  extensions: Search Firestore with Typesense will be granted the following access to project typesense-extension-testground
- Cloud Datastore User (Provides read/write access to data in a Cloud Datastore database. Intended for application developers and service accounts.)
? Do you wish to continue? Yes
? Please enter a new name for this instance: firestore-typesense-search-3f2u
i  extensions: answer the questions below to configure your extension:

Firestore Collection Path: The Firestore collection that needs to be indexed into Typesense.
? Enter a value for Firestore Collection Path: books

Firestore Collection Fields (Optional): A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields.
? Enter a value for Firestore Collection Fields:

Typesense Hosts: A comma-separated list of Typesense Hosts. For single node clusters, a single hostname is sufficient. For multi-node Highly Available or SDN Clusters, please be sure to mention all hostnames.
? Enter a value for Typesense Hosts: test

Typesense API Key: An Typesense API key with admin permissions. Click on "Generate API Key" in cluster dashboard in Typesense Cloud
? Where would you like to store your secrets? You must select at least one value Google Cloud Secret Manager
? This secret will be stored in Cloud Secret Manager (https://cloud.google.com/secret-manager/pricing) as ext-firestore-typesense-search-3f2u-TYPESENSE_API_KEY and managed by Firebase Extensions (Firebase Ex
tensions Service Agent will be granted Secret Admin role on this secret).
Enter a value for Typesense API Key: [hidden]
✔  secretmanager: Granted roles/secretmanager.admin on projects/1053026563937/secrets/ext-firestore-typesense-search-3f2u-TYPESENSE_API_KEY to service-1053026563937@gcp-sa-firebasemods.iam.gserviceaccount.com

Typesense Collection Name: Typesense collection name to index data into
? Enter a value for Typesense Collection Name: books_firestore

Cloud Functions location: Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the location selection guide.
? Which option do you want enabled for this parameter? Select an option with the arrow keys, and use Enter to confirm your choice. You may only select one option. Iowa (us-central1)

⠸ Installing your extension instance. This usually takes 3 to 5 minutes...

I wonder if the params.env file needs to be formatted in a specific way for secrets manager? Could you share your current params config file (masking sensitive data)?

[jon-flowers]

@jasonbosco generally my file looks like this:

FIRESTORE_COLLECTION_PATH=receiptTemplates FIRESTORE_COLLECTION_FIELDS= TYPESENSE_HOSTS=abc.a1.typesense.net TYPESENSE_API_KEY=abc123 TYPESENSE_COLLECTION_NAME=eng_test_receiptTemplates LOCATION=useast1

The TYPESENSE_API_KEY value (filled with a generic string above) is just the actual API key that I get directly from the Typesense cloud console. This definitely worked before, but if I need to change the format I am fine with that.

Were you able to get the above results by using a params.env file with the same formatting? Or is that the interactive flow where you entered the API key in when it prompted for it?

[i14h]

@jon-flowers are you using the non-interactive flow --force option?

[jon-flowers]

@jon-flowers are you using the non-interactive flow --force option?

Yes I am

[i14h]

@joehan fyi

[i14h]

we think it's a combination of specifying the secret in the params.env file and using the --force param. But the error message should've been more clear. Let us investigate a little bit.

[jasonbosco]

Thanks @i14h.

@jon-flowers While this is being investigated, could you use v0.2.7 of the extension, which doesn't use Secret Manager:

firebase ext:install typesense/firestore-typesense-search@0.2.7 --project=projectId --params=params.env --force

[jon-flowers]

@i14h got it, thank you!

@jasonbosco that makes sense, I will do that and that should unblock me for now. Appreciate the help!

[i14h]

I strongly suggest using the more secure version. Try not specifying your secret in the params.env file and see if that makes the problem go away? We don't suggest storing secrets in plain text files that could end up checked into your source repo

[jon-flowers]

@i14h Sorry I probably should have added more context here.

I am using Terraform (with the third-party Typesense provider, as well as Google and local-execs) to build out a Typesense collection, alias, enable the extension for an existing Firestore database, and create the backfill sync document.

Everything is pretty straightforward EXCEPT installing this extension, because the Google/Firebase providers do not have a Terraform resource for that. So I am using a local-exec instead. Though I am actually using two local-execs:

1. The first one builds a simple params.env file, using dynamic values. Long-term, these are stored in either an env.yaml (for non-secret parameters) or an env-sensitive.yaml (for secret parameters). The env.yaml is maintained in the Terragrunt source code, the sensitive yaml is maintained in S3 and retrieved every time Terragrunt runs. The params.env file is built within Terragrunt and so is only maintained temporarily in the cache before being deleted.
2. The second local-exec using the built params.env file to run the Firebase CLI command that installs the extension for my given Firebase/Firestore app/database.

I need to add the Secret in there somehow because otherwise the CLI asks for it to be input, and I need this to be non-interactive. I am using Terragrunt/Terraform to deploy this to 20+ environments (and maintain those going forward), so i am trying to avoid having to manually run CLI commands for each environment, especially as we continue to add more in the future. I know that if I don't specify the secret it will ask me for it (and the flow works, with it correctly converting the API key to a secret), but this will not work in CircleCI/Terragrunt

[Salakar]

You should be able to create the secret manually on the Google Cloud Console in secrets manager or through the API then reference in the .env file like so:

TYPESENSE_API_KEY=projects/${param:PROJECT_NUMBER}/secrets/<secretName>/versions/latest

[jon-flowers]

You should be able to create the secret manually on the Google Cloud Console in secrets manager or through the API then reference in the .env file like so:

TYPESENSE_API_KEY=projects/${param:PROJECT_NUMBER}/secrets/<secretName>/versions/latest

I think I tried something similar to that, I agree it should work but if I remember correctly the problem was that the extension is not expecting that format; basically it seems like the extension at least is expecting the API key itself in one place and the GCP format you used above in another place, meaning it fails no matter what. Assuming I am remembering this correctly, when I tested it using the above method I got a different error that was due to the formatting.

This sort of makes sense, since the UI / manual flow to enable the extension actually does ask for first the API Key itself, then prompts you to click a button to create the secret, and then uses that secret in the config. So when you use the CLI method to enable the extension it's probably trying to do at least two things:

1. Using TYPESENSE_API_KEY to generate a new secret
2. Using that new secret in the config

And unfortunately it's not possible to format TYPESENSE_API_KEY in a way that works for both those steps.

[i14h]

@joehan and @pavelgj can you take a look into this?

[PMLS3]

Has anyone got a solution for this? First time using the extension and also getting Request failed with HTTP code 404 | Server said: Not Found... I've set mine up through the Firebase Web Ui...

[jasonbosco]

@PMLS3 This issue is referencing an HTTP 400 error, when installing this extension via the Firebase CLI.

Your HTTP 404 error is one that comes from Typesense and is unrelated to this issue. You're seeing that error most likely because you need to setup a collection in Typesense first - the extension does not create the Typesense collection for you. See Step 3 here. If that's not the issue, and you do have a collection setup in Typesense, could you open a separate issue?