Project cannot be created because you have exceeded your allotted project quota.

[imranzunzani]

We are using the Deployment Manager to create projects from a dedicated Project for DM under an Org Account. From time to time we keep hitting the following error:

project cannot be created because you have exceeded your allotted project quota.\n type:
message: '{"ResourceType":"cloudresourcemanager.v1.project","ResourceErrorCode":"429","ResourceErrorMessage":"The
location: /deployments

We have tried changing the Service Account and this doesn't go away unless we switch to another project (or create a new one). This is strange, as under the 'Quotas' we don't see anything that has exceeded the cap. Could you please help?

[ocsig]

Project quota is the number of projects you are allowed to create within your organization. This is a bit more special then resource quotas you see under the quota page. Partially because this quota is tied to your Org/Billing account, also because deletion of the project has a 30 day grace period

At the end of the 30-day period, the project and all of its resources are deleted and cannot be recovered. Until it is deleted, the project will count towards your project quota.

I recommend you to follow the project quota increase procedures: Request Billing Quota Increase

Further explanation about project quotas.

[imranzunzani]

We are aware of that and that quota is not exceeded. We can create a project through gcloud create project

The problem is only when using Deployment Manager.

@ocsig You closed it too quick. Could you please reopen it?

[ocsig]

Are you able to create a new project with gcloud even if you are in the same project where you have the issue with DM?

[imranzunzani]

Yes

[ocsig]

The reply what you see is coming from the underlying API, let me escalate this to the DM product team. Would you mind to specify what type are you using for project creation?

[imranzunzani]

The 'type' in the config is set to 'project.py'.

[imranzunzani]

The config yaml we are using as below:

imports:
- path: project.py

resources:
  # The "name" property below will be the ID of the new project
  # If you want your project to have a different name, use the "project-name"
  # property.
  - name: 'project_name'
    type: project.py
    properties:
        # Change this to your organization ID.
        organization-id: 'org_id'
        # You can also create the project in a folder.
        # If both organization-id and parent-folder-id are provided,
        # the project will be created in parent-folder-id.
        parent-folder-id: 'folder_id'

        # Change the following to your organization's billing account
        billing-account-name: billingAccounts/'account_id'

        # The apis to enable in the new project.
        # To see the possible APIs, use: gcloud services list --available
        apis:
          - compute.googleapis.com
          - deploymentmanager.googleapis.com
          - pubsub.googleapis.com
          - storage-component.googleapis.com
          - monitoring.googleapis.com
          - logging.googleapis.com
          - stackdriver.googleapis.com
          - sourcerepo.googleapis.com
          - containerregistry.googleapis.com
          - cloudbuild.googleapis.com
          - container.googleapis.com
          - servicenetworking.googleapis.com

        # The service accounts you want to create in the project
        service-accounts:
          - 'sa-name'

        # Makes the service account that Deployment Manager would use in the
        # generated project when making deployments in this new project a
        # project owner.
        set-dm-service-account-as-owner: true

        # The patches to apply to the project's IAM policy. Note that these are
        # always applied as a patch to the project's current IAM policy, not as a
        # diff with the existing properties stored in DM. This means that removing
        # a binding from the 'add' section will not remove the binding on the
        # project during the next update. Instead it must be added to the 'remove'
        # section.
        iam-policy-patch:
            # These are the bindings to add.
            add:
              - role: roles/owner
                members:
                  - group:'google-group-account'
              - role: roles/viewer
                members:
                  - 'viewer-sa'
              - role: roles/editor
                members: 
                  - 'editor-sa'

[ocsig]

If you are using https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/community/cloud-foundation/templates/project implementation, please change to our new repo, all of our templates got a large upgrade over the summer: https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/master/dm/templates/project/project.py

NOTE: This update contains several changes which are backwards incompatible!

Would you mind to confirm your project.py is using type': 'gcp-types/cloudresourcemanager-v1:projects' ?

[imranzunzani]

Yes. It is using 'cloudresourcemanager-v1:projects'.

It is present as: 'type': 'cloudresourcemanager.v1.project'

Is there a difference because of the hyphen (-)?

[ocsig]

Please be aware that 'type': 'cloudresourcemanager.v1.project' is not equal to type': 'gcp-types/cloudresourcemanager-v1:projects'. Would you mind to try to use type': 'gcp-types/cloudresourcemanager-v1:projects' and let us know if you face the same issue? It helps our debugging.

There are historical reasons for the two different type: https://cloud.google.com/deployment-manager/docs/configuration/supported-resource-types https://cloud.google.com/deployment-manager/docs/configuration/supported-gcp-types (In this case they are calling the same API endpoints behind the scenes, but there can be input mapping and other differences between the two type.

You can see the latest project template implementation with documentation and examples here.

[imranzunzani]

Tried with the new repo. Still a 429.

ERROR: build step 0 "gcr.io/sharedservices-controlplane/provisioner" failed: exit status 1
ERROR

\ exceeded\"}"
\ <eye3 title='RESOURCE_EXHAUSTED'/> generic::RESOURCE_EXHAUSTED: project quota\
:\"429\",\"ResourceErrorMessage\":\"project quota exceeded\\ncom.google.apps.framework.request.StatusException:\
message: "{\"ResourceType\":\"gcp-types/cloudresourcemanager-v1:projects\",\"ResourceErrorCode\"\
location: /deployments/self-driving-truck-app/resources/self-driving-truck-app-project
- code: RESOURCE_ERROR
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1575299713992-598ba0bde0837-87a52e22-adfd604f]: errors:
........failed.

Is there a quota for the Service Account that the Deployment Manager uses? If yes, could you please tell us where to check that and how to increase it?

[ocsig]

We are investigating this case together with input from other customers. In case if you have the appropriate support access, you can open a ticket with GCP Supports for better tracking.

[imranzunzani]

Let me check and update about the support access, as this got handed over to other team mates now. However, we have switched those configurations to Terraform from Deployment Manager, due to this issue.

[ocsig]

The issue was related to Quota limitation which is tied to the SA, not just the Organization. An other workaround is to use BYOSA ( Bring your own Service Account, currently in alpha), so you can use any SA with deployment manager, just like with Terraform.

[imranzunzani]

Thanks for clearing this. It kept us confused for long. Is this then expected to go away with a fix, in future? And the non-default SA/BYOSA won't exhibit this because of no limit, or is there a limit anyway, just higher? May we know if there are limits, what are those and why do we have them for this scenario?