Issue with using Cloud DNS in Deployment Manager

droid76 commented 4 years ago

I am trying to assign a unique hostname to my Compute Engine instance by creating a Managed DNS Zone in my template. I have tried following the steps mentioned here to create it But when I try to create the deployment using the gcloud command, I get the following error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1596237072244-5abc4e7ab1d45-dcb5be0c-d264db02]: errors:
- code: RESOURCE_ERROR
  location: /deployments/deployment1/resources/create-dns
  message: '{"ResourceType":"gcp-types/dns-v1:dns.changes.create","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Request
    contains an invalid argument.","status":"INVALID_ARGUMENT","statusMessage":"Bad
    Request","requestPath":"https://dns.googleapis.com/dns/v1/projects/[PROJECT_NAME]/managedZones/managed-zone/changes","httpMethod":"POST"}}'

This is how my template looks like:

   {
        'name':'managed-zone',
        'type':'dns.v1.managedZone',
        'properties':
        {
            'description':'Managed zone',
            'dnsName':'headnode.example.com.'
        }
    },
    {
        'type': 'compute.v1.globalAddress',
        'name': 'global-ip',
        'properties': 
        {
            'description': 'Global IP address',
        }
    },
    {
        'name':'create-dns',
        'action':'gcp-types/dns-v1:dns.changes.create',
        'metadata':
        {
            'runtimePolicy':['CREATE']
        },
        'properties':
        {
             'managedZone':'managed-zone',
             'additions':
             [
                 {
                     'name':'dm.headnode.example.com',
                     'type':'A',
                     'ttl':3600,
                     'rrdatas': ["$(ref.global-ip.address)"]
                 }
             ]
        }
    },
    {
        'name':'delete-dns',
        'action':'gcp-types/dns-v1:dns.changes.create',
        'metadata':
        {
            'runtimePolicy':['DELETE']
        },
        'properties':
        {
             'managedZone':'managed-zone',
             'deletions':
             [
                 {
                     'name':'dm.headnode.example.com',
                     'type':'A',
                     'ttl':3600,
                     'rrdatas': ["$(ref.global-ip.address)"]
                 }
             ]
        }
    },

Does anyone have any suggestions on how to solve this?

ocsig commented 4 years ago

Hi @droid76 ,

Please avoid using this old example and actions wherever it is possible. GCP-Types are supporting DNS records since this article was written, please take a look at this sample: google/resource-snippets/dns-v1/one_a_record.jinja Please also check the other files in the same folder. Let me know if this did not help to solve your issue!

droid76 commented 4 years ago

Hi @ocsig,

I used gcp-types/dns-v1:resourceRecordSets as mentioned in the sample, but I still get the exact same error. This is how my template looks like now:

 {
        'name':'managed-zone',
        'type':'gcp-types/dns-v1:managedZones',
        'properties':
        {
            'description':'Managed zone',
            'dnsName':'headnode.example.com.'
        }
    },
    {
        'type': 'compute.v1.globalAddress',
        'name': 'global-ip',
        'properties': 
        {
            'description': 'Global IP address'
        }
    },
    {
        'name':'resource-record-sets',
        'type':'gcp-types/dns-v1:resourceRecordSets',
        'properties':
        {
             'managedZone':'managed-zone',
             'records':
             [
                 {      
                    'name':'dm.headnode.example.com',
                     'type':'A',
                     'ttl':3600,
                     'rrdatas': ["$(ref.global-ip.address)"]
                 }
             ]
        }
    },

ocsig commented 4 years ago

I would use a reference to the Managed Zone:

managedZone: $(ref.managed-zone.name)

This way you get a proper error is there is a typo. I don't think this is the issue now.

Can you find the actual API call in the logs whick tries to create the dns record and post the reques body, url and the response? ( so we can spot the ivalid part?)

droid76 commented 4 years ago

Could you tell me where I can find the logs? I tried looking under my deployment in Google Cloud console but couldn't find it

ocsig commented 4 years ago

Audit logs Logs viewer

You can construct complex search queries, but the easiest is to look for the resource name or something else which should be unique.

droid76 commented 4 years ago

These are the logs for the latest deployment of Managed Zone:

[
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "status": {
        "code": 3
      },
      "authenticationInfo": {
        "principalEmail": "[SERVICE_ACCOUNT]",
        "serviceAccountDelegationInfo": [
          {
            "firstPartyPrincipal": {
              "principalEmail": "cloud-dm@prod.google.com"
            }
          }
        ]
      },
      "requestMetadata": {
        "callerIp": "66.102.8.24",
        "requestAttributes": {
          "time": "2020-08-04T06:09:49.608Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "dns.googleapis.com",
      "methodName": "dns.changes.create",
      "authorizationInfo": [
        {
          "permission": "dns.changes.create",
          "granted": true,
          "resourceAttributes": {}
        }
      ],
      "resourceName": "managedZones/managed-zone",
      "request": {
        "@type": "type.googleapis.com/cloud.dns.api.ChangesCreateRequest",
        "change": {
          "additions": [
            {
              "ttl": 3600,
              "type": "A",
              "name": "dm.headnode.example.com",
              "rrdata": [
                "35.190.65.187"
              ]
            }
          ]
        },
        "managedZone": "managed-zone",
        "project": "[PROJECT_ID]"
      },
      "response": {
        "@type": "type.googleapis.com/cloud.dns.api.ChangesCreateResponse"
      }
    },
    "insertId": "c7m5aee22pbc",
    "resource": {
      "type": "dns_managed_zone",
      "labels": {
        "location": "global",
        "project_id": "[PROJECT_ID]",
        "zone_name": "managed-zone"
      }
    },
    "timestamp": "2020-08-04T06:09:49.566Z",
    "severity": "ERROR",
    "logName": "projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity",
    "receiveTimestamp": "2020-08-04T06:09:50.135758538Z"
  }
]

I couldn't figure out the error by just looking at the logs. Could the error be because I am using references to the managed zone and global IP address while creating the resource record set?

ocsig commented 4 years ago

That is what I wanted to check:

"resourceName": "managedZones/managed-zone",
      "request": {
        "@type": "type.googleapis.com/cloud.dns.api.ChangesCreateRequest",
        "change": {
          "additions": [
            {
              "ttl": 3600,
              "type": "A",
              "name": "dm.headnode.example.com",
              "rrdata": [
                "35.190.65.187"
              ]
            }
          ]
        }

This clarifies that DM resolves the references and passess the right values to the DNS API.

I believe the error is your DNS name, which needs a . at the end: "name": "dm.headnode.example.com.", to be FQDN valid.

Would you mind to add that and let me know if this solves the error?

droid76 commented 4 years ago

I added the "." at the end of the name. But when I try to create the deployment, I get this error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1596553741857-5ac0ea2a5f35a-bb89ebc4-d6ba5c63]: errors:
- code: INTERNAL_ERROR
  message: "Internal error. Please try again or contact Google Support. (Code: '0')"

I am able to see the resource record created in the managed zone though. Screenshot (87) Also, when I try to delete the deployment, the managed zone fails to be deleted as it contains a resource recordset. Is there a way I can get the resource record set to be deleted when the deployment is deleted without having to manually delete it?

droid76 commented 4 years ago

Could not figure out reason for the error so abandoned this method and found another way to get the public hostname of the instance

trevorjwilliams commented 3 years ago

I ran into this same problem, and can reproduce it with the following configuration:

resources:
- name: dns-zone
  type: gcp-types/dns-v1:managedZones
  properties:
    name: imma-dns-zone
    dnsName: fake.domain.com.
    visibility: public
    # "description" is required, but can be blank.
    description: ""
- name: dns-records
  type: gcp-types/dns-v1:resourceRecordSets
  properties:
    name: imma-dns-record
    managedZone: $(ref.dns-zone.name)
    records:
    - name: fake.domain.com.
      type: A
      ttl: 300
      rrdatas:
      - 1.2.3.4

Specifically, I'm observing the following behavior:

Creating a deployment with this configuration returns an error, but both the zone and the DNS entry are successfully created.
Reviewing the audit logs, all of the DNS API invocations seem to have completed successfully. The first warning in the logs is from DM:

    {
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 3,
      "message": "BAD_REQUEST"
    },
    "authenticationInfo": {
      "principalEmail": "(redacted)"
    },
    "requestMetadata": {
      "callerIp": "68.41.222.190",
      "callerSuppliedUserAgent": "google-cloud-sdk gcloud/310.0.0 command/gcloud.deployment-manager.deployments.create invocation-id/a56d98f8f71f4ee7bd58820ee1ba00e5 environment/None environment-version/None interactive/True from-script/False python/3.6.8 term/screen (Macintosh; Intel Mac OS X 19.6.0),gzip(gfe),gzip(gfe)"
    },
    "serviceName": "deploymentmanager.googleapis.com",
    "methodName": "v2.deploymentmanager.deployments.insert",
    "resourceName": "projects/(redacted)/global/deployments/dnstest",
    "request": {
      "@type": "type.googleapis.com/deploymentmanager.deployments.insert"
    }
  },
  "insertId": "4g3ga8ddois",
  "resource": {
    "type": "deployment",
    "labels": {
      "name": "dnstest",
      "project_id": "(redacted)"
    }
  },
  "timestamp": "2020-09-17T00:43:44.606Z",
  "severity": "ERROR",
  "logName": "projects/(redacted)/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "operation-1600303396605-5af77ab7cce09-bb015d41-53c72a2c",
    "producer": "deploymentmanager.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2020-09-17T00:43:45.302243830Z"
}

Attempting to delete the deployment fails, but the audit logs include two errors in the DNS API calls as well as an error from DM:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 9
    },
    "authenticationInfo": {
      "principalEmail": "(redacted)@cloudservices.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "cloud-dm@prod.google.com"
          }
        }
      ]
    },
    "requestMetadata": {
      "callerIp": "74.125.210.123",
      "requestAttributes": {
        "time": "2020-09-17T00:53:10.887Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "dns.googleapis.com",
    "methodName": "dns.managedZones.delete",
    "authorizationInfo": [
      {
        "permission": "dns.managedZones.delete",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceName": "managedZones/imma-dns-zone",
    "request": {
      "project": "(redacted)",
      "@type": "type.googleapis.com/cloud.dns.api.ManagedZonesDeleteRequest",
      "managedZone": "imma-dns-zone"
    },
    "response": {
      "@type": "type.googleapis.com/cloud.dns.api.ManagedZonesDeleteResponse"
    }
  },
  "insertId": "n1hwoae6vcu4",
  "resource": {
    "type": "dns_managed_zone",
    "labels": {
      "project_id": "(redacted)",
      "zone_name": "imma-dns-zone",
      "location": "global"
    }
  },
  "timestamp": "2020-09-17T00:53:10.815Z",
  "severity": "ERROR",
  "logName": "projects/(redacted)/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2020-09-17T00:53:11.696421036Z"
}

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 9
    },
    "authenticationInfo": {
      "principalEmail": "(redacted)@cloudservices.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "cloud-dm@prod.google.com"
          }
        }
      ]
    },
    "requestMetadata": {
      "callerIp": "74.125.210.125",
      "requestAttributes": {
        "time": "2020-09-17T00:53:11.450Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "dns.googleapis.com",
    "methodName": "dns.managedZones.delete",
    "authorizationInfo": [
      {
        "permission": "dns.managedZones.delete",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceName": "managedZones/imma-dns-zone",
    "request": {
      "project": "(redacted)",
      "managedZone": "imma-dns-zone",
      "@type": "type.googleapis.com/cloud.dns.api.ManagedZonesDeleteRequest"
    },
    "response": {
      "@type": "type.googleapis.com/cloud.dns.api.ManagedZonesDeleteResponse"
    }
  },
  "insertId": "77rsp1e6vnsa",
  "resource": {
    "type": "dns_managed_zone",
    "labels": {
      "zone_name": "imma-dns-zone",
      "location": "global",
      "project_id": "(redacted)"
    }
  },
  "timestamp": "2020-09-17T00:53:11.379Z",
  "severity": "ERROR",
  "logName": "projects/(redacted)/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2020-09-17T00:53:12.416680665Z"
}

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 3,
      "message": "BAD_REQUEST"
    },
    "authenticationInfo": {
      "principalEmail": "(redacted)"
    },
    "requestMetadata": {
      "callerIp": "2002:a05:6834:30f::",
      "callerSuppliedUserAgent": "CloudConsoleService Google-API-Java-Client Google-HTTP-Java-Client/1.26.0-SNAPSHOT (gzip)"
    },
    "serviceName": "deploymentmanager.googleapis.com",
    "methodName": "v2beta.deploymentmanager.deployments.delete",
    "resourceName": "projects/(redacted)/global/deployments/dnstest",
    "request": {
      "@type": "type.googleapis.com/deploymentmanager.deployments.delete"
    }
  },
  "insertId": "-wqmphyd73z4",
  "resource": {
    "type": "deployment",
    "labels": {
      "name": "dnstest",
      "project_id": "(redacted)"
    }
  },
  "timestamp": "2020-09-17T00:53:14.193Z",
  "severity": "ERROR",
  "logName": "projects/(redacted)/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "operation-1600303973431-5af77cdde79a0-94f289f2-e8d23203",
    "producer": "deploymentmanager.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2020-09-17T00:53:15.127864230Z"
}

I ended up using a configuration that looked a lot like the first post in this thread as a workaround.

Also, it looks like some of the DNS DM samples in this repo are missing the name field in their dns records.

GoogleCloudPlatform / deploymentmanager-samples

Issue with using Cloud DNS in Deployment Manager #567