Constraint constraints/cloudfunctions.allowedIngressSettings violated for projects/project_id attempting CreateFunctionActionV1 with ingress_settings set to INGRESS_SETTINGS_UNSPECIFIED

[RajvirBains]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 1.22 (latest)
• Configuration values (redact info where appropriate):
  • _
  • _

[REQUIRED] Step 3: Describe the problem

The default setting for "Ingress setting" in the extension seems to be not set. So, if an organization is restricting "Ingress settings = Allow all traffic", then the installation fails. I am not sure what are the actual requirements for this extension are i.e. does it really need "Ingress settings = Allow all traffic".

Steps to reproduce:

If your organization policy applies constraints/cloudfunctions.allowedIngressSettings and doesn't allow "Ingress settings = Allow all traffic" for CloudFunctions, then the installation of this extension fails with the below error.

RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export/resources/fsexportbigquery: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"The request has violated one or more Org Policies. Please refer to the respective violations for more information.","status":"FAILED_PRECONDITION","details":[{"@type":"type.googleapis.com/google.rpc.PreconditionFailure","violations":[{"type":"constraints/cloudfunctions.allowedIngressSettings","subject":"orgpolicy:projects/project_id","description":"**Constraint constraints/cloudfunctions.allowedIngressSettings violated for projects/project_id attempting CreateFunctionActionV1 with ingress_settings set to INGRESS_SETTINGS_UNSPECIFIED**. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information."}]}],"statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/project_id/locations/northamerica-northeast1/functions","httpMethod":"POST"}}

Expected result

Installation should work successfully.

Actual result

But it fails.

Recommendation: If the extension doesn't need "Ingress settings = Allow all traffic", then I think it would be good to set the default to "Allow internal traffic only".

[dackers86]

Hi @RajvirBains.

Thanks for reporting the issue.

I've added to our project board for investigation.

[i14h]

@pavelgj I think this is something not specific to the bigquery extension and we set it on all extensions. WDYT? Can we restrict this?

[pavelgj]

We actually don't set ingressSettings and that is the problem when org policies require them to be specified. It's most likely not an issue specific to bigquery extension. I'm not sure if we can support anything other than ALLOW_ALL though. We currently don't support VPC, and I'm not sure whether ALLOW_INTERNAL_AND_GCLB is an option.

[RajvirBains]

@pavelgj from the standpoint of the needs of this extension, do you know if the it triggers the CloudFunction using the HTTP endpoint and if it needs to be open to the Internet? It might be good to try out your suggestion ALLOW_INTERNAL_AND_GCLB. Based on your assessment "Allow internal traffic only" is not an option? correct?

[i14h]

@RajvirBains the point is there's no way to set these per extension on the backend. The solution would be for us to enable setting these in your project which you can't right now but we're discussing what can be done. I don't think this'll be available anytime soon though because of the huge backlog we have.

[pavelgj]

@RajvirBains Can you tell me if your org policy constraints/cloudfunctions.allowedIngressSettings includes ALLOW_ALL? (If you're not comfortable sharing this info here, you can file a support ticket with details and reference this github issue). I'm just curious if the org policy just requires ingress settings to be explicitly set, then we can easily explicitly set to ALLOW_ALL and hopefully it will quickly resolve your issue. But if your org policy does not allow ALLOW_ALL, then I'm afraid the solution would be blocked on extensions supporting VPC and I don't have a timeline for that yet.

[RajvirBains]

Hi @pavelgj we disallow ALLOW_ALL setting in our Org Policy. I am ok for now as I was able to get an exemption for this project. But I think it is good to have this on your roadmap to resolve properly. I fully understand the backlog issue, so as far as it is on the list of recognized issues, it is good. I find that this extension is very helpful and over time, your team will need to iron out issues like this for enterprise customers. Thanks for your prompt response on this issue.

[RajvirBains]

@pavelgj see if your team can try doing this security assessment for me on this topic. My Security Team flagged "Ingress settings = Allow all traffic" as a security risk and my explanation to them is as below. Their concern is that these functions can be exploited over the internet. See if I have it right.

The CloudFunctions Trigger Type of HTTP allows access over the internet, but in case of this Firebase extension Trigger Type is "Firestore" and as we trigger CREATE, UPDATE, DELETE events in Firestore, they get served by CloudFunctions, which then send data to BQ. These CloudFunctions don't have HTTP endpoints at all. So, for this extension there is no risk of external access, it is just that the default setting of "Ingress settings = Allow all traffic" when CloudFunctions are created, is not correct.

[pavelgj]

@RajvirBains I think you are correct -- for 1st gen cloud functions currently used by Firebase Extensions there is no public HTTP endpoint created for event triggers (ex. firestore triggers used by bigquery export extension). However, technically the platform does allow extensions to deploy functions with public HTTP(s) endpoints and there are extensions which do use this feature. If your security team does allow all ingress traffic then they will not have a way to prevent you from deploying those extensions and violate their policy.

Have you considered trying to manually deploy the bigquery export function with VPC connector? It's technically a single function (with a firestore trigger). The source code is here: https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/functions/src/index.ts It will need only minor tweaks to the trigger definition boilerplate.

[RajvirBains]

@pavelgj wonder if your link is correct https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/functions/src/index.ts? I didn't see any trigger definition in this. Could you please confirm?

So, you are suggesting that I download your [firestore-bigquery-export] code, tweak it to change the trigger definition i.e. just change it so that "Ingress settings = Allow internal traffic only" and deploy it using the Firebase command-line? This might be an acceptable alternative and would really appreciate if I can get some help with this.

[pavelgj]

@RajvirBains The cloud functions trigger definitions are in the code.

Here is the sample code changes and instructions for how to manually deploy the bigquery export function with a VPC connector (I haven't tested the VPC part, just copy pasted from documentation, so you may need to tweak it a bit): https://github.com/pavelgj/extensions/commit/e9e9942d430afb5a00908a003337e614c114f64a

[RajvirBains]

@pavelgj looks like you forgot to paste some content here.

[RajvirBains]

@pavelgj disregard my previous comment. Sorry about that, missed the link at the end.

[RajvirBains]

@pavelgj, POC worked like a charm. Your instructions were very helpful. Only change I had to make was to replace your VPC Connection code with just an ingress.

Original provide by you:

exports.fsexportbigquery = functions.runWith({
    vpcConnector: 'test-connector',
    vpcConnectorEgressSettings: 'PRIVATE_RANGES_ONLY'

Changed to:

exports.fsexportbigquery = functions.runWith({
    ingressSettings: 'ALLOW_INTERNAL_ONLY'

I was able to deploy to a project that had constraints/cloudfunctions.allowedIngressSettings Org Policy on it and was able to confirm that CloudFunction deploys, triggers and copies data successfully to the BQ table.

I do see that if I now maintain my own code base to do this one-liner, I have some work to do to maintain my home grown system. The CloudFunction deploys with " fsexportbigquery" name and I couldn't figure out how to change it. Also, with these instructions I couldn't figure out how I can deploy multiple extensions i.e. many .env files. Well, I am not a TS/JS expert and also wasn't able to use a variable here .document("mycollection/{doc}") and hard-coded it as per your instructions.

Anyways, you know where I am leading with this :) Would it be possible for your team to consider this change as an enhancement? I think there is lot of security value here.

[pavelgj]

Hmm... for some reason I wrongly assumed that for ingressSettings: 'ALLOW_INTERNAL_ONLY' it required a VPC connector. This may be a much simpler fix than I expected. Let me investigate whether it's safe to just always set it to ALLOW_INTERNAL_ONLY for functions used for event triggers. Obviously we will still need to set ALLOW_ALL for http endpoints and you won't be able to install those extensions (bigquery export is not one of them), and that is exactly what we want.

[RajvirBains]

@pavelgj see if you can provide an update on this. We have ticket open with Google Support and our TAM, so will keep everyone informed.

[pavelgj]

I'm running some e2e tests. I can't provide a firm ETA, but this will take at least a couple of weeks to confirm and release. I'll provide another update ones I have testing completed.

[RajvirBains]

Sounds good @pavelgj. Thanks for the update.

[RajvirBains]

@pavelgj is this change still looking on track? Do you think we will have it by the end of next week or so?

[pavelgj]

Everything is in place, just not broadly enabled (still testing a few scenarios). If you can share your project id (through the support ticket) I can enable the ingress settings feature for your project early.

[RajvirBains]

Thanks Pavel. I have updated the support case (existing ticket for this issue) and have asked them to share the project ids with you. I have provided reference to this Github Issue also.

[pavelgj]

@RajvirBains your projects are allowlisted, you can give it a try.

[RajvirBains]

@pavelgj this worked out great from the Firebase console and it successfully deployed a cloudfunction with "Ingress settings = Allow internal traffic only". I tested one replication testcase to ensure that it is working end to end. Good work.

[eslamkarim]

hi @pavelgj do we have any estimates or timelines for this fix to be deployed?

[pavelgj]

@eslamkarim, the fix is fully rolled out now.

[harpreetpaul]

Hi,

We are facing the similar issue when deploying the BQ Firebase extension. I am part of the same org as @RajvirBains. What was the final solution to this problem.

Here is the error Constraint constraints/cloudfunctions.allowedIngressSettings violated for projects/<project-id> attempting CreateFunctionActionV1 with ingress_settings set to ALLOW_ALL.