setIamPolicy for Cloud Functions

[dinvlad]

Hi Team,

I tried to set IAM policy bindings (invoker etc.) for gcp-types/cloudfunctions-v1:projects.locations.functions resources. However, DM complains with No method found to update access control on resource 'my-function' of type 'cloudfunctions-v1'.

Looking at the type provider, it indeed doesn't seem to include setIamPolicy method.

I then defined my own my-project/cloudfunctions-v1:projects.locations.functions custom type provider, and in that case setIamPolicy method works. However, I'm then unable to update my functions resource, for an unrelated reason (i.e. DM can't find the "patch" method because the path parameter is named differently in the API between "patch" and "create"... I'm not sure why that is, given that the "native" gcp-types provider works for updates).

As a result, I'm not sure how we can set IAM policy through DM. Another possible workaround is to do it inside a "utility" cloud function, and gcp-types/cloudfunctions-v1:cloudfunctions.projects.locations.functions.call that function from the template to set the policy on the "target" function. But even then we're in a bit of a catch-22, where we also need to set IAM policy on the "utility" function..

I should also mention that Terraform currently doesn't support setting IAM policy on a Function resource either, so we're a bit out of options when it comes to declarative deployment.

What would be a good way to proceed here? Thanks a lot

[ocsig]

Hi,

Can you clarify, how did you tried to call the setIamPolicy?

Here is the old implementation of project level setIamPolicy: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/ocsig-patch-2/community/cloud-foundation/templates/iam_member/iam_member.py

It needs 3 actions:

'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.getIamPolicy', # Get
'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy',  # Create/update
'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy', # Delete 

NOTE: This is not a nice or best practice implementation. We are working on removing the need of using the actions with the virtual type providers. You can see the new implementations under projects: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/iam_member/iam_member.py

I'm not sure why that is, given that the "native" gcp-types provider works for updates

The gcp-types are based on the discovery doc, the same way as you created the type provider. However in some cases when the underlying API is not 100% aligned to the CRUD structure DM expects there are overwrites defined. Here is some info about advanced type provider options: https://cloud.google.com/deployment-manager/docs/configuration/type-providers/advanced-configuration-options

Let me know if this helped, I do a deeper look what is the status of the virtual endpoint for function IAM binding.

Last but not least please use the new repo, we implemented significant updates on every template: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/

[ocsig]

I looked up and the virtual.*.iamMemberBinding endpoint implementation is currently under internal review. I am reaching out the team to push this through fast. Hopefully it can be live within a few weeks.

[dinvlad]

Thanks @ocsig, does that endpoint address resource-based IAM policies?

To clarify, instead of setting a project-wide IAM policy I wanted to grant a list of members access to invoke specific GCFs. Normally (for other types of resources, like buckets), this is done through accessControl key, which in turn calls setIamPolicy method on the resource. But in case of projects.locations.functions resource from gcp-types, that method doesn’t appear to be exposed yet in the DM type definitions (even though it’s present on the API).

Could you perhaps update the gcp-types provider with this method? I defined my own provider as an exact copy of gcp-types one, plus setIamPolicy input mapping, and that enabled accessControl in DM. But as I mentioned, the update/patch method then stops working, which appears to be due to some internal override in gcp-types that’s not exposed through DM public API. I believe this is due to the requirement by DM to have exactly the same path parameter names for create/update methods, which is not the case for GCF API. I stumbled on the same issue trying to configure a provider for Cloud Run - everything works (incl. setIamPolicy), except for updates.

With that said, I’ve now converted this stack to Terraform, which has just added IAM bindings for GCFs and also offers nice-to-haves like ability to easily upload source ZIPs to GCS for GCF deployment. I think we’ll go with that stack for now.

[ocsig]

Custom type: In my previous comment I quoted the methodMap which you need to add to your custom type provider, that solves the update/patch problem.

IAM: I did not suggested to use project wide IAM, but to use the same action based implementation like it was before in https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/ocsig-patch-2/community/cloud-foundation/templates/iam_member/iam_member.py This is an alternative option to the custom type provider for the short time, while you are waiting for the virtual.*.iamMemberBinding rollout.

As I mentioned, virtual.*.iamMemberBinding is in the pipeline, which will include resources like CloudFunctions.

Upload: Please take a look at our latest implementation example, it is supporting file upload for months: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/cloud_function/examples/cloud_function_upload.yaml Full template: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/cft-dm-dev/dm/templates/cloud_function

[dinvlad]

In my previous comment I quoted the methodMap which you need to add to your custom type provider, that solves the update/patch problem.

Thanks @ocsig - apologies that I missed that. I'll try to see if that solves the problem.

Please take a look at our latest implementation example, it is supporting file upload for months

Yep, was aware of that - however, it's a bit more cumbersome to do it this way (i.e. through Cloud Build) than to simply use a google_storage_bucket_object in TF.

A bit OT here, but TF also offers other niceties like being able to call Berglas executable locally to grant GCF access to secrets. Overall it appears to fit our current use case better atm (but we do have another stack that's working well in DM and we don't see a reason to switch..)

[dinvlad]

As a follow-up on methodMap - is there any official documentation on the use of that parameter? I could not find it in the advanced guide.

[ocsig]

@dinvlad: I may started the explanation from the wrong end. There is a large codebase ( for DM and for TF) called Cloud Foundation Toolkit. The DM templates moved out recently to their own repo. It covers many of the early usecases and provides you reference implementation. It is expected you to use this as an external library in your codebase, not re-implementing it.

About CF uploads. As of today, this is a fundamental difference between DM and TF. DM runs on a GCP managed server where you can't run custom commands like gsUtil, this makes it harder to implement features like file upload. The team is currently implementing local expansion, which means you will be able to communicate with your backed services and possibly call local commands like upload files etc. Stay tuned.

The methodMap is currently missing from the public documentation.

[dinvlad]

Sounds good, thanks for the explanation! I'm looking forward for any updates. I'll go ahead and close this for now.

[ocsig]

Note: I removed methodMap from my previous comment. That feature never reached the type-provider APIs external production state. ( It's not available publicly.)

[dinvlad]

Understood, thanks! I'll be looking forward to any official news and alphas.

[mr-pascal]

Hey, is there any udpate when (if ever) the setIamPolicy will be available for GCF? Just run into the same problem. I can deploy my GCF via DM without problems and it works like a charm, but if I can't set the accessControl via my configuration file, its rather useless for me to use DM if it doesn't support something like this..

[ocsig]

Hi @Abszissex I did not update this thread, but the functionality went live a few months ago:

https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates/iam_member

Or you can use the virtual type directly: gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding

I realized we published tests for CloudFunctions, but no example. I will fix that soon.

[mr-pascal]

Hey @ocsig ,

first of all thanks for the very fast response. I hope you maybe can help me with my specific problem. Maybe you can also use parts for it for the example? So I have a deployment.yml file which is passed to the DM via the --config flag. This file imports a .jinja file. Everything works fine except the IAM stuff. So for testing I wanted to add allUsers the invoker role. But everytime I do the DM returns with a 404. If the functionality was added that much time ago, shouldn't it work then? Funny thing is that the DM runs without errors when I rename gcpIamPolicy to policy even though it still doesn't add the policy to the GCF, it just doesn't error. Also the same happens when I move the accessControl section from the .jinja file to the deployment.yml file => no error, but no IAM is added.

I mean if you say it was added months ago, I of course trust you, I just don't get it why it doesn't work? Where is my error?

Thanks in Advance

// configurable_functions.jinja
resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
  name: {{ env['deployment'] }}-function
  properties:
    parent: projects/{{ env['project'] }}/locations/{{ properties['region'] }}
    function: {{ env['deployment'] }}
    sourceArchiveUrl: gs://{{ properties['codeBucket'] }}/{{ properties['codeBucketObject'] }}
    entryPoint: {{ properties['entryPoint'] }}
    runtime: {{ properties['runtime'] }}
    availableMemoryMb: {{ properties['availableMemoryMb'] }}
    timeout: {{ properties['timeout'] }}
    httpsTrigger: {}
  accessControl:
    gcpIamPolicy:
      bindings:
      - members:
        - allUsers
        role: roles/cloudfunctions.invoker

// deployment.yml
imports:
  - path: configurable_functions.jinja
resources:
  - name: get-user
    type: configurable_functions.jinja
    properties:
      codeBucket: get-user
      codeBucketObject: WILL_BE_OVERWRITTEN_ANYWAY.zip
      region: europe-west1
      entryPoint: handler
      runtime: nodejs8
      availableMemoryMb: 256
      timeout: 60s

// The Error when I try to update my deployment with the IAM policy

gcloud deployment-manager deployments update get-user --config dist/apps/get-user/deployment.yml
The fingerprint of the deployment is Ew4-vG1IdUU2xQ05JlsPJg==
Waiting for update [operation-1589395598673-5a58c00110577-2e67e76b-a3fa6f0d]...
...............................................failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1589395598673-5a58c00110577-2e67e76b-a3fa6f0d]: errors:
- code: RESOURCE_ERROR
  location: /deployments/get-user/resources/get-user-function
  message: '{"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"404","ResourceErrorMessage":{"statusMessage":"Not
    Found","requestPath":"https://cloudfunctions.googleapis.com/v1/:setIamPolicy","httpMethod":"POST"}}'

Finished with ERROR!!

Edit: I just got it to work (after a lot of research + reverse engineerign) like I want, even though I would prefer to simply set the IAM binding via the accessControl property, so it can be easy set on the function itself (seems cleaner to me)

my solution

resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
  name: {{ env['deployment'] }}-function
  properties:
    parent: projects/{{ properties['projectId'] }}/locations/{{ properties['region'] }}
    function: {{ env['deployment'] }}-function
    sourceArchiveUrl: gs://{{ properties['codeBucket'] }}/{{ properties['codeBucketObject'] }}
    entryPoint: {{ properties['entryPoint'] }}
    runtime: {{ properties['runtime'] }}
    availableMemoryMb: {{ properties['availableMemoryMb'] }}
    timeout: {{ properties['timeout'] }}
    httpsTrigger: {}

- name: my-iam-binding-func-allUsers
  type: gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding
  properties:
    resource: projects/{{ properties['projectId'] }}/locations/{{ properties['region'] }}/functions/{{ env['deployment'] }}-function
    role: roles/cloudfunctions.invoker
    member: allUsers

[saiaman]

any news on this issue ??? it's a relly big deal

[dinvlad]

@saiaman I think the solution described by @ocsig and @Abszissex works already, IIRC

[saiaman]

Only Works after first deploy not the first

[saiaman]

And this is not a solution: Google have to patch their deployment manager that’s all

[sjvanrossum]

@saiaman The solution mentioned above works, although I'd recommend using a reference to create a dependency between the two resources.

Here's a sample (Jinja template) from our test suite for the virtual IAM member binding resource:

resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
  name: {{ env['deployment'] }}-my-function
  properties:
    parent: projects/{{ env['project'] }}/locations/{{ properties['region'] }}
    function: my-{{ env['deployment'] }}
    sourceArchiveUrl: gs://cloud-function-sample/function.zip
    entryPoint: handler
    runtime: nodejs8
    httpsTrigger: {}
- name: {{ env['deployment'] }}-my-function-iam
  type: gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding
  properties:
    resource: $(ref.{{ env['deployment'] }}-my-function.name)
    member: serviceAccount:{{ env['project_number'] }}@cloudservices.gserviceaccount.com
    role: roles/cloudfunctions.viewer