Closed dinvlad closed 4 years ago
Hi,
Can you clarify, how did you tried to call the setIamPolicy?
Here is the old implementation of project level setIamPolicy: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/ocsig-patch-2/community/cloud-foundation/templates/iam_member/iam_member.py
It needs 3 actions:
'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.getIamPolicy', # Get
'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy', # Create/update
'action': 'gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy', # Delete
NOTE: This is not a nice or best practice implementation. We are working on removing the need of using the actions with the virtual type providers. You can see the new implementations under projects: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/iam_member/iam_member.py
I'm not sure why that is, given that the "native" gcp-types provider works for updates
The gcp-types are based on the discovery doc, the same way as you created the type provider. However in some cases when the underlying API is not 100% aligned to the CRUD structure DM expects there are overwrites defined. Here is some info about advanced type provider options: https://cloud.google.com/deployment-manager/docs/configuration/type-providers/advanced-configuration-options
Let me know if this helped, I do a deeper look what is the status of the virtual endpoint for function IAM binding.
Last but not least please use the new repo, we implemented significant updates on every template: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/
I looked up and the virtual.*.iamMemberBinding endpoint implementation is currently under internal review. I am reaching out the team to push this through fast. Hopefully it can be live within a few weeks.
Thanks @ocsig, does that endpoint address resource-based IAM policies?
To clarify, instead of setting a project-wide IAM policy I wanted to grant a list of members access to invoke specific GCFs. Normally (for other types of resources, like buckets), this is done through accessControl
key, which in turn calls setIamPolicy
method on the resource. But in case of projects.locations.functions
resource from gcp-types
, that method doesn’t appear to be exposed yet in the DM type definitions (even though it’s present on the API).
Could you perhaps update the gcp-types
provider with this method? I defined my own provider as an exact copy of gcp-types
one, plus setIamPolicy
input mapping, and that enabled accessControl
in DM. But as I mentioned, the update/patch method then stops working, which appears to be due to some internal override in gcp-types
that’s not exposed through DM public API. I believe this is due to the requirement by DM to have exactly the same path parameter names for create/update methods, which is not the case for GCF API. I stumbled on the same issue trying to configure a provider for Cloud Run - everything works (incl. setIamPolicy), except for updates.
With that said, I’ve now converted this stack to Terraform, which has just added IAM bindings for GCFs and also offers nice-to-haves like ability to easily upload source ZIPs to GCS for GCF deployment. I think we’ll go with that stack for now.
Custom type:
In my previous comment I quoted the methodMap
which you need to add to your custom type provider, that solves the update/patch problem.
IAM: I did not suggested to use project wide IAM, but to use the same action based implementation like it was before in https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/ocsig-patch-2/community/cloud-foundation/templates/iam_member/iam_member.py This is an alternative option to the custom type provider for the short time, while you are waiting for the virtual.*.iamMemberBinding rollout.
As I mentioned, virtual.*.iamMemberBinding is in the pipeline, which will include resources like CloudFunctions.
Upload: Please take a look at our latest implementation example, it is supporting file upload for months: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/cft-dm-dev/dm/templates/cloud_function/examples/cloud_function_upload.yaml Full template: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/cft-dm-dev/dm/templates/cloud_function
In my previous comment I quoted the methodMap which you need to add to your custom type provider, that solves the update/patch problem.
Thanks @ocsig - apologies that I missed that. I'll try to see if that solves the problem.
Please take a look at our latest implementation example, it is supporting file upload for months
Yep, was aware of that - however, it's a bit more cumbersome to do it this way (i.e. through Cloud Build) than to simply use a google_storage_bucket_object
in TF.
A bit OT here, but TF also offers other niceties like being able to call Berglas executable locally to grant GCF access to secrets. Overall it appears to fit our current use case better atm (but we do have another stack that's working well in DM and we don't see a reason to switch..)
As a follow-up on methodMap
- is there any official documentation on the use of that parameter? I could not find it in the advanced guide.
@dinvlad: I may started the explanation from the wrong end. There is a large codebase ( for DM and for TF) called Cloud Foundation Toolkit. The DM templates moved out recently to their own repo. It covers many of the early usecases and provides you reference implementation. It is expected you to use this as an external library in your codebase, not re-implementing it.
About CF uploads. As of today, this is a fundamental difference between DM and TF. DM runs on a GCP managed server where you can't run custom commands like gsUtil, this makes it harder to implement features like file upload. The team is currently implementing local expansion, which means you will be able to communicate with your backed services and possibly call local commands like upload files etc. Stay tuned.
The methodMap is currently missing from the public documentation.
Sounds good, thanks for the explanation! I'm looking forward for any updates. I'll go ahead and close this for now.
Note: I removed methodMap from my previous comment. That feature never reached the type-provider APIs external production state. ( It's not available publicly.)
Understood, thanks! I'll be looking forward to any official news and alphas.
Hey, is there any udpate when (if ever) the setIamPolicy will be available for GCF? Just run into the same problem. I can deploy my GCF via DM without problems and it works like a charm, but if I can't set the accessControl via my configuration file, its rather useless for me to use DM if it doesn't support something like this..
Hi @Abszissex I did not update this thread, but the functionality went live a few months ago:
https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates/iam_member
Or you can use the virtual type directly:
gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding
I realized we published tests for CloudFunctions, but no example. I will fix that soon.
Hey @ocsig ,
first of all thanks for the very fast response. I hope you maybe can help me with my specific problem. Maybe you can also use parts for it for the example?
So I have a deployment.yml
file which is passed to the DM via the --config
flag. This file imports a .jinja
file. Everything works fine except the IAM stuff. So for testing I wanted to add allUsers
the invoker role. But everytime I do the DM returns with a 404.
If the functionality was added that much time ago, shouldn't it work then? Funny thing is that the DM runs without errors when I rename gcpIamPolicy
to policy
even though it still doesn't add the policy to the GCF, it just doesn't error. Also the same happens when I move the accessControl
section from the .jinja
file to the deployment.yml
file => no error, but no IAM is added.
I mean if you say it was added months ago, I of course trust you, I just don't get it why it doesn't work? Where is my error?
Thanks in Advance
// configurable_functions.jinja
resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
name: {{ env['deployment'] }}-function
properties:
parent: projects/{{ env['project'] }}/locations/{{ properties['region'] }}
function: {{ env['deployment'] }}
sourceArchiveUrl: gs://{{ properties['codeBucket'] }}/{{ properties['codeBucketObject'] }}
entryPoint: {{ properties['entryPoint'] }}
runtime: {{ properties['runtime'] }}
availableMemoryMb: {{ properties['availableMemoryMb'] }}
timeout: {{ properties['timeout'] }}
httpsTrigger: {}
accessControl:
gcpIamPolicy:
bindings:
- members:
- allUsers
role: roles/cloudfunctions.invoker
// deployment.yml
imports:
- path: configurable_functions.jinja
resources:
- name: get-user
type: configurable_functions.jinja
properties:
codeBucket: get-user
codeBucketObject: WILL_BE_OVERWRITTEN_ANYWAY.zip
region: europe-west1
entryPoint: handler
runtime: nodejs8
availableMemoryMb: 256
timeout: 60s
// The Error when I try to update my deployment with the IAM policy
gcloud deployment-manager deployments update get-user --config dist/apps/get-user/deployment.yml
The fingerprint of the deployment is Ew4-vG1IdUU2xQ05JlsPJg==
Waiting for update [operation-1589395598673-5a58c00110577-2e67e76b-a3fa6f0d]...
...............................................failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1589395598673-5a58c00110577-2e67e76b-a3fa6f0d]: errors:
- code: RESOURCE_ERROR
location: /deployments/get-user/resources/get-user-function
message: '{"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"404","ResourceErrorMessage":{"statusMessage":"Not
Found","requestPath":"https://cloudfunctions.googleapis.com/v1/:setIamPolicy","httpMethod":"POST"}}'
Finished with ERROR!!
Edit: I just got it to work (after a lot of research + reverse engineerign) like I want, even though I would prefer to simply set the IAM binding via the accessControl property, so it can be easy set on the function itself (seems cleaner to me)
my solution
resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
name: {{ env['deployment'] }}-function
properties:
parent: projects/{{ properties['projectId'] }}/locations/{{ properties['region'] }}
function: {{ env['deployment'] }}-function
sourceArchiveUrl: gs://{{ properties['codeBucket'] }}/{{ properties['codeBucketObject'] }}
entryPoint: {{ properties['entryPoint'] }}
runtime: {{ properties['runtime'] }}
availableMemoryMb: {{ properties['availableMemoryMb'] }}
timeout: {{ properties['timeout'] }}
httpsTrigger: {}
- name: my-iam-binding-func-allUsers
type: gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding
properties:
resource: projects/{{ properties['projectId'] }}/locations/{{ properties['region'] }}/functions/{{ env['deployment'] }}-function
role: roles/cloudfunctions.invoker
member: allUsers
any news on this issue ??? it's a relly big deal
@saiaman I think the solution described by @ocsig and @Abszissex works already, IIRC
Only Works after first deploy not the first
And this is not a solution: Google have to patch their deployment manager that’s all
@saiaman The solution mentioned above works, although I'd recommend using a reference to create a dependency between the two resources.
Here's a sample (Jinja template) from our test suite for the virtual IAM member binding resource:
resources:
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
name: {{ env['deployment'] }}-my-function
properties:
parent: projects/{{ env['project'] }}/locations/{{ properties['region'] }}
function: my-{{ env['deployment'] }}
sourceArchiveUrl: gs://cloud-function-sample/function.zip
entryPoint: handler
runtime: nodejs8
httpsTrigger: {}
- name: {{ env['deployment'] }}-my-function-iam
type: gcp-types/cloudfunctions-v1:virtual.projects.locations.functions.iamMemberBinding
properties:
resource: $(ref.{{ env['deployment'] }}-my-function.name)
member: serviceAccount:{{ env['project_number'] }}@cloudservices.gserviceaccount.com
role: roles/cloudfunctions.viewer
Hi Team,
I tried to set IAM policy bindings (invoker etc.) for
gcp-types/cloudfunctions-v1:projects.locations.functions
resources. However, DM complains withNo method found to update access control on resource 'my-function' of type 'cloudfunctions-v1'
.Looking at the type provider, it indeed doesn't seem to include
setIamPolicy
method.I then defined my own
my-project/cloudfunctions-v1:projects.locations.functions
custom type provider, and in that casesetIamPolicy
method works. However, I'm then unable to update myfunctions
resource, for an unrelated reason (i.e. DM can't find the "patch" method because the path parameter is named differently in the API between "patch" and "create"... I'm not sure why that is, given that the "native"gcp-types
provider works for updates).As a result, I'm not sure how we can set IAM policy through DM. Another possible workaround is to do it inside a "utility" cloud function, and
gcp-types/cloudfunctions-v1:cloudfunctions.projects.locations.functions.call
that function from the template to set the policy on the "target" function. But even then we're in a bit of a catch-22, where we also need to set IAM policy on the "utility" function..I should also mention that Terraform currently doesn't support setting IAM policy on a Function resource either, so we're a bit out of options when it comes to declarative deployment.
What would be a good way to proceed here? Thanks a lot