Private VPC Peering for Cloud SQL is not supported by deployment manager

[Priyankasaggu11929]

(TL;DR) Solution: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/issues/549#issuecomment-613842019

Basically, there is no way to connect/enable servicenetworking.googleapis.com service for a vpc network through deployment-manager.

The gcloud command for the above action/operation is:

gcloud services vpc-peerings connect --service=servicenetworking.googleapis.com --ranges=<my-range> --network=<my-network> --project=<my-project>

Any pointers for how this could be done via deployment-manager?

[ocsig]

You can create your custom-type provider based on the Service Networking API. You can use an action for the services.connections.create endpoint and pass the appropriate parameters. I think the create would work fairly okey. Update ( patch ) may need some input mapping. I do not see any delete function in the API.

Alternatively here is a solution which involves DM starting an Instance, Instance calling gcloud and then cleanup: https://stackoverflow.com/questions/56908506/using-gcloud-services-vpc-peerings-connect-in-deployment-manager

[Priyankasaggu11929]

@ocsig this time I tried to read about creating a custom type-provider. I still didn't understand how you wrote that options file.

Although I came across an alternative way to create it using gcloud command. I want to confirm, is it doing the same thing?

The command is gcloud alpha deployment-manager type-providers create test-custom-type --descriptor-url='https://servicenetworking.googleapis.com/$discovery/rest?version=v1'

[ocsig]

Ho @Priyankasaggu11929,

TL:DR; yes.

This page describes how to create a custom type. The options file you reference has two parts. Part one is the authentication token. That you can copy past as it is. ( The link above also explains that.) The input mapping part is only needed if the API is unconcetional, like you need to put additional info to tha path, instead of the body of the requests.

In you case I don't think you need input mapping. Use a simple options file with the authentication section. Use gcloud beta deployment-manager type-providers create test-custom-type --descriptor-url='https://servicenetworking.googleapis.com/$discovery/rest?version=v1' --api-options-file options.yaml.

[Priyankasaggu11929]

So, the options.yaml file will include this much only:

options:
    inputMappings:
        - fieldName: Authorization
          location: HEADER
          value: $.concat("Bearer ", $.googleOauth2AccessToken())
          methodMatch: .*

And could you explain this a lil bit more.

You can use an action for the services.connections.create endpoint and pass the appropriate parameters.

[ocsig]

Options.yaml: yes.

Action: If you use type in GCP it will try to do Create, update, delete of the resource when you Create, update, delete the deployment. For this API, there is only a create and a patch endpoint, for now lets focus on create:

resources:
  - name: 'addpeering'
    action: 'my-project-ID/test-custom-type:services.connections.create'
    properties:

I don't have time right now to provide you a full wokring example, let me know how you are progressing.

[Priyankasaggu11929]

@ocsig this much is a good overview. I'll update you here with the further steps.

Thank you so much :)

[Priyankasaggu11929]

@ocsig,

When I tried using the custom type-provider like this: my-project-ID/test-custom-type:services.connections.create

It gave the following error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1585304228648-5a1d36768d64f-860de874-4e97a0ee]: errors:
- code: COLLECTION_NOT_FOUND
  message: Collection 'services.connections.create' not found in discovery doc 'https://servicenetworking.googleapis.com/$discovery/rest?version=v1'

Then I looked into the discovery document, and replaced services.connections.create with servicenetworking.services.connections.create. The above error got resolved, but I got a new one for a missing properties field parent.

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1585304392334-5a1d3712a7ab8-250e9a08-2450a4e5]: errors:
- code: CONDITION_NOT_MET
  location: /deployments/annn/resources/sample-peering-createPeer->$.properties
  message: |
    error: object has missing required properties (["parent"])
        level: "error"
        schema: {"loadingURI":"#","pointer":"/create"}
        instance: {"pointer":""}
        domain: "validation"
        keyword: "properties"
        required: ["parent"]
        missing: ["parent"]

I want to understand, where I can find what necessary fields should be mentioned while creating a resource through this custom type-provider.

[OT: If I have multiple deployment templates say A, B and C, in one infra stack. And resource created by template A is dependent on the one created by template B. How can I mention the template A resource in the dependsOn metadata field of template B resouce.

Like, in the above case, I need a vpc network to be created first before connecting it to the above service. So, one way is via creating both the resources in one template file. But if in general there are 2 jinja template files, is there any way to import resources from one jinja template inside another? ]

[Priyankasaggu11929]

Also, like with gcloud, it is possible to describe the configs set for the resources deployed on GCP. For example, gcloud sql instances describe test-postgres describes a cloud sql instance and returns json configs .

Is there any way to describe a VPC network peering like this in the console?

[ocsig]

Dependencies Unfortunately template level dependsOn is not supported currently in DM. There is 2 workaround here.

1. using references. If a resource has a property, even if it is just a label, with a reference in it to an other resource, it will be dependent on it.
2. using dependsOn

For both of these, your template needs to make sure that this is set in every resource within that template. For example a template can have a dependsOn PROPERTY, which is then passed to every resource within as a dependsOn metadata. We used this strategy in the Cloud Foundation Toolkit tempaltes.

Missing Fields Looking up the API documentation ( or the Discovery Doc) is your best source to know what properties you need to send. The parent property is required, but it is in the PATH, not in the BODY of the requests. Please add a parent property to your resource and let me know if it works. If not, we need to extend the type definition with something like this:

options:
    inputMappings:
        - fieldName: Authorization
          location: HEADER
          value: $.concat("Bearer ", $.googleOauth2AccessToken())
          methodMatch: .*
    collectionOverrides:
    - collection: services.connections
       options:
        virtualProperties: |
          schema: http://json-schema.org/draft-04/schema#
          type: object
          properties:
            parent:
              type : string
          required:
          - parent
        inputMappings:
        - fieldName: location
          location: PATH
          methodMatch: ^create$
          value: $.resource.properties.parent

[Priyankasaggu11929]

@ocsig I think I am a little lost here. I am trying to create this service as a separate resource which I am in doubt as if it is the right way to do it or not.

[My approach]:

1. I checked the discovery document for the API and saw it requires only one parameter i.e Parent.

But with only Parent, it threw a resource field missing error, which I think is obvious here as I need to mention a network here.

So, I wrote the jinja template as:

resources:
  - name: addpeering
    action: {project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create
    properties:
      parent: services/servicenetworking.googleapis.com
      network: projects/{project-id}/global/networks/dev-network-02  

With that, I got this error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1585311677577-5a1d523667adf-fe2b269c-ead0edb1]: errors:
- code: RESOURCE_ERROR
  location: /deployments/anl/resources/addpeering
  message: '{"ResourceType":"{project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"The
    caller does not have permission","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://servicenetworking.googleapis.com/v1/services/servicenetworking.googleapis.com/connections","httpMethod":"POST"}}'

I've just started with DevOps and writing IaC, so really sorry for sounding very noob.

I will clarify my use-case a bit here:

1. I am trying to create a private Postgres Cloud SQL instance.

2. With UI, when I try to choose Private IP and then select a network vpc, It shows a button Allocate and connect.

3. This button automatically creates the VPC Network Peering for the respective vpc network.

I want to achieve this automatic VPC Network Peering creation through the DM templates.

[ocsig]

You are doing a good progress. First of all appologies, you are on uncharted territories. ( At least under documented.)

Yes, parent, network are mandatory fields.

You are getting a Permission error, which gives me the impression that your call is correct, reaching the API backend.

A little bit guessing here, your DM service account ( [PROJECT_NUMBER]@cloudservices.gserviceaccount.com ) is missing servicenetworking.services.addPeering permission in the parent project. This can be added by giving roles/compute.networkAdmin to the DM Service account on the parent project.

[Priyankasaggu11929]

I am looking at the permissions right now.

Would you mind checking this question once please https://github.com/GoogleCloudPlatform/deploymentmanager-samples/issues/549#issuecomment-604939088

[Priyankasaggu11929]

Yes, setting up the roles/compute.networkAdmin permission solved the Permission error.

With the follow-up runs, I needed to add two more fields, so now the templates look like:

resources:
  - name: addpeering
    action: {project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create
    properties:
      parent: services/servicenetworking.googleapis.com
      network: projects/{project-id}/global/networks/dev-network-02
      peering: cloudsql-postgres-googleapis-com
      reservedPeeringRanges:
      - google-managed-services-default

and with this, the error now is:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1585318457546-5a1d6b78497ea-593c3503-16223b13]: errors:
- code: RESOURCE_ERROR
  location: /deployments/aph/resources/addpeering
  message: "{\"ResourceType\":\"{project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create\"\
    ,\"ResourceErrorCode\":\"412\",\"ResourceErrorMessage\":\"allocated IP range 'google-managed-services-default'\
    \ not found in network\"}"

I took some reference from here for what should be the values for the new fields.

[ocsig]

Also, like with gcloud, it is possible to describe the configs set for the resources deployed on GCP. For example, gcloud sql instances describe test-postgres describes a cloud sql instance and returns json configs .

Is there any way to describe a VPC network peering like this in the console?

gcloud compute networks peerings list gcloud compute networks peerings list-routes gcloud compute networks vpc-access connectors describe I found these, is that what you are looking for?

[ocsig]

Yes, setting up the roles/compute.networkAdmin permission solved the Permission error.

With the follow-up runs, I needed to add two more fields, so now the templates look like:

resources:
  - name: addpeering
    action: atlanhq/vpcpeering-v1beta-type:servicenetworking.services.connections.create
    properties:
      parent: services/servicenetworking.googleapis.com
      network: projects/atlanhq/global/networks/dev-network-02
      peering: cloudsql-postgres-googleapis-com
      reservedPeeringRanges:
      - google-managed-services-default

and with this, the error now is:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1585318457546-5a1d6b78497ea-593c3503-16223b13]: errors:
- code: RESOURCE_ERROR
  location: /deployments/aph/resources/addpeering
  message: "{\"ResourceType\":\"{project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create\"\
    ,\"ResourceErrorCode\":\"412\",\"ResourceErrorMessage\":\"allocated IP range 'google-managed-services-default'\
    \ not found in network\"}"

I took some reference from here for what should be the values for the new fields.

Can you run gcloud beta compute addresses list and see your named range?

[Priyankasaggu11929]

Can you run gcloud beta compute addresses list and see your named range?

I ran the command but it doesn't contain the named range I created.

[ocsig]

That is align with the DM Error. We need to make sure google-managed-services-default range is created in the right project/network.

[Priyankasaggu11929]

The above screenshot does contain google-managed-services-default range but not in the network that I am trying to create it in using the template. But this range is in the right project though.

[Priyankasaggu11929]

I am using the below template for creating postgres cloud sql instance. As I mentioned in this comment here, that I am trying to achieve the automatic VPC Network Peering creation through the DM templates, could you please take a look at the template and see if something could be done for the same.

{% set deployment_name = env['deployment']  %}
{% set instance_name = deployment_name + '-instance'  %}
{% set database_name = deployment_name + '-db'  %}
{% set ID = env['deployment'] + '-' + env['name'] %}

resources:

- name: {{ ID }}-master
  type: gcp-types/sqladmin-v1beta4:instances
  properties:
    region: {{ properties['region'] }}  
    backendType: {{ properties['backendType'] }}  
    gceZone: {{ properties['gceZone'] }}  
    instanceType: CLOUD_SQL_INSTANCE
    databaseVersion: {{ properties['databaseVersion'] }}  
    settings:
      tier: {{ properties['tier'] }}
      activationPolicy: ALWAYS
      availabilityType: ZONAL
      backupConfiguration:
        enabled: true
        pointInTimeRecoveryEnabled: false
        replicationLogArchivingEnabled: false
      dataDiskSizeGb: {{ properties['dataDiskSizeGb'] }}
      dataDiskType: {{ properties['dataDiskType'] }}      
      ipConfiguration:
        authorizedNetworks:
        - kind: sql#aclEntry
          name: {{ properties["ipConfiguration"]["authorizedNetworks"]["name"] }}
          value:  {{ properties["ipConfiguration"]["authorizedNetworks"]["value"] }}
        ipv4Enabled: true
        privateNetwork: projects/{project-id}/global/networks/{{ properties["ipConfiguration"]["privateNetwork"] }}

- name: {{ ID }}-db
  type: gcp-types/sqladmin-v1beta4:databases
  properties:
    name: {{ properties["database"]["name"] }}
    instance: $(ref.{{ ID }}-master.name)
    charset: {{ properties["database"]["charset"] }}

- name: {{ ID }}-db-root
  type: gcp-types/sqladmin-v1beta4:users
  properties:
    name: {{ properties["database-root"]["user"] }}
    instance: $(ref.{{ ID }}-master.name)
    host: ""
    password: {{ properties["database-root"]["password"] }}
  metadata:
    dependsOn:
    - {{ ID }}-db

[ocsig]

The above screenshot does contain google-managed-services-default range but not in the network that I am trying to create it in using the template. But this range is in the right project though.

Your range has to be in the RIGHT network. So please create the range there. Potentially use a unique name. ( default refers to the default network.)

[ocsig]

I am using the below template for creating postgres cloud sql instance. As I mentioned in this comment here, that I am trying to achieve the automatic VPC Network Peering creation through the DM templates, could you please take a look at the template and see if something could be done for the same.

I am a bit lost with this comment. You need the action discussed in this thread as resource in your template ( and the peering range has to exist in that network):

  - name: addpeering
    action: {project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create
    properties:
      parent: services/servicenetworking.googleapis.com
      network: projects/{project-id}/global/networks/{network-id}
      peering: cloudsql-postgres-googleapis-com
      reservedPeeringRanges:
      - google-managed-services-{network-id}

[Priyankasaggu11929]

@ocsig Thank you so much, the above change in the reservedPeeringRanges value name google-managed-services-{network-id} solved the entire issue.

I achieved Postgres private-only deployment automation by incorporating the above template.

[jjlorenzo]

@Priyankasaggu11929 can you post your working example? Thanks

[Priyankasaggu11929]

@jjlorenzo

The jinja template, say example-postgres.jinja will include the following:

{% set ID = env['name'] %}

resources:

######## Network ###########

- name: {{ ID }}-network
  type: compute.v1.network
  properties:
    autoCreateSubnetworks: false

######### SUBNETS ##########

{% for i in range(properties["ipCidrRange"]|length) %}
- name: {{ ID }}-subnet-{{ i }}
  type: compute.v1.subnetwork
  properties:
    network: $(ref.{{ ID }}-network.selfLink)
    privateIpGoogleAccess: true
    ipCidrRange: {{ properties["ipCidrRange"][i] }}
    region: {{ properties["region"] }}
    logConfig:
      aggregationInterval: {{ properties["log"]["aggregationInterval"] }}
      flowSampling: {{ properties["log"]["flowSampling"] }}
      enable: true
{% endfor %}

######### GOOGLE MANAGED SERVICES ##########

- name: addpeering
  action: {project-id}/vpcpeering-v1beta-type:servicenetworking.services.connections.create
  properties:
    parent: services/servicenetworking.googleapis.com
    network: projects/{project-id}/global/networks/{{ ID }}-network
    peering: cloudsql-postgres-googleapis-com
    reservedPeeringRanges:
    - google-managed-services-{{ ID }}-network
  metadata:
    dependsOn:
    - {{ ID }}-network

- name: google-managed-services-{{ ID }}-network
  type: compute.beta.globalAddress
  properties:
    network: $(ref.{{ ID }}-network.selfLink)
    purpose: VPC_PEERING
    addressType: INTERNAL
    prefixLength: 16

# ########## POSTGRES CREATION ##########

- name: {{ ID }}-master
  type: gcp-types/sqladmin-v1beta4:instances
  properties:
    region: {{ properties['region'] }}  
    backendType: {{ properties['backendType'] }}  
    gceZone: {{ properties['gceZone'] }}  
    instanceType: CLOUD_SQL_INSTANCE
    databaseVersion: {{ properties['databaseVersion'] }}  
    settings:
      tier: {{ properties['ptier'] }}
      activationPolicy: ALWAYS
      availabilityType: ZONAL
      backupConfiguration:
        enabled: true
        pointInTimeRecoveryEnabled: false
        replicationLogArchivingEnabled: false
      dataDiskSizeGb: {{ properties['dataDiskSizeGb'] }}
      dataDiskType: {{ properties['dataDiskType'] }}      
      ipConfiguration:
        authorizedNetworks:
        - kind: sql#aclEntry
          name: {{ properties["ipConfiguration"]["authorizedNetworks"]["name"] }}
          value: {{ properties["ipConfiguration"]["authorizedNetworks"]["value"] }}
        ipv4Enabled: true
        privateNetwork: projects/{project-id}/global/networks/{{ ID }}-network
  metadata:
    dependsOn:
    - {{ ID }}-network
    - addpeering

- name: {{ ID }}-db
  type: gcp-types/sqladmin-v1beta4:databases
  properties:
    name: {{ properties["database"]["name"] }}
    instance: $(ref.{{ ID }}-master.name)
    charset: {{ properties["database"]["charset"] }}

- name: {{ ID }}-db-root
  type: gcp-types/sqladmin-v1beta4:users
  properties:
    name: {{ properties["database-root"]["user"] }}
    instance: $(ref.{{ ID }}-master.name)
    host: ""
    password: {{ properties["database-root"]["password"] }}
  metadata:
    dependsOn:
    - {{ ID }}-db

And the config.yaml will include the following:

imports:
- path: example-postgres.jinja

resources:

- name: demo
  type: example-postgres.jinja
  properties:
    region: us-west1
    zone: us-west1-a

###### VPC CONFIGS ###### 

    ipCidrRange:
    - 172.16.0.0/21
    log:
      aggregationInterval: INTERVAL_10_MIN
      flowSampling: 0.5

###### POSTGRES CONFIGS ######

    backendType: SECOND_GEN
    gceZone: us-west1-a
    databaseVersion: POSTGRES_11
    ptier: db-custom-2-7680
    dataDiskSizeGb: '10'
    dataDiskType: PD_SSD
    ipConfiguration:
      authorizedNetworks:
        name: example
        value: {ip-range}
    database:
      name: test
      charset: utf8
    database-root:
      user: root
      password: password 

You need to change value for the following:

1. project-id
2. Inside config.yaml,
   • change name & value in ipConfiguration -> authorizedNetworks.
   • And pass user and password values accordingly in database-root section.

Hope this helps!

[jjlorenzo]

@Priyankasaggu11929 thanks you very much.

[andytheapedemontague]

Just to say thanks @Priyankasaggu11929 and @ocsig this was super useful and saved me loads of time. Looking forward to Google supporting this in Deployment Manager directly. In the meantime this works well.

[huy-nguyen]

Hi @ocsig, @Priyankasaggu11929 and @andytheapedemontague, I'm working on this same problem. My approach is to create the type provider with this command:

gcloud beta deployment-manager \
type-providers create service-networking-v1 \
--descriptor-url='https://servicenetworking.googleapis.com/$discovery/rest?version=v1' \
--api-options-file=service-networking-type-provider.yaml \
--project project-id

where service-networking-type-provider.yaml has the following content:

options:
  inputMappings:
    - fieldName: Authorization
      location: HEADER
      value: $.concat("Bearer ", $.googleOauth2AccessToken())
      methodMatch: .*

collectionOverrides:
- collection: services.connections
  options:
    virtualProperties: |
      schema: http://json-schema.org/draft-04/schema#
      type: object
      properties:
        networkName:
          type : string
      required:
      - networkName

    inputMappings:
      - fieldName: parent
        location: PATH
        methodMatch: ^(create|list)$
        value: concat("services/", $.resource.properties.service)
      - fieldName: network
        location: BODY
        methodMatch: ^create$
        value: concat("projects/", $.project, "/global/networks/", $.resource.properties.networkName)
      - fieldName: network
        location: QUERY
        methodMatch: ^list$
        value: concat("projects/", $.project, "/global/networks/", $.resource.properties.networkName)

With this type definition, my resource definition looks like this:

resources:
- name: my-peering-connection
  type: project-id/service-networking-v1:services.connections   
  properties:
    service: servicenetworking.googleapis.com
    reservedPeeringRangaes:
    - previously-set-up-peering-range-name
    networkName: previously-set-up-network-name

However, because my type provider definition doesn't have a working GET method, deployment will always fail because deployment manager will try to create the resource on every "update" and fail because the resource already exists (due to being created on the first run). I notice that your solution uses action instead of type. Can any of you explain the difference to me? Thanks!

[ocsig]

Action: This is an alpha feature, generally discouraged to use it, however it does the job for you. ( Only running once at creation and not trying to do get calls.) An action is successful if gets a 200 reply, but does not query the underlying resource.

[Bobirmirzo]

I am sorry when I use the above code, I am getting following error:

• code: RESOURCE_NOT_FOUND message: The type [/vpcpeering-v1beta-type:servicenetworking.services.connections.create] was not found.

I do not know why