Create cluster success in UI but got ResourceErrorCode

[carolynhu]

Hi there, I encountered similar issue as this: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/issues/148

I'm using https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/examples/v2/gke/python/cluster.py and based on my recent changes https://github.com/GoogleCloudPlatform/deploymentmanager-samples/pull/536 to create a GKE cluster.

My cluster is created successfully as least on the UI part.

But in the terminal, I got the following errors:

 Waiting for create [operation-1581380209089-59e41c59e1bfb-c62501e0-2a81feab]...
failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1581380209089-59e41c59e1bfb-c62501e0-2a81feab]: errors:
- code: RESOURCE_ERROR
  location: /deployments/carolyn-manager/resources/carolyn-manager-3-cluster-py-type-apps
  message: "{\"ResourceType\":\"deploymentmanager.v2beta.typeProvider\",\"ResourceErrorCode\"\
    :\"DESCRIPTOR_URL_FETCH_ERROR\",\"ResourceErrorMessage\":\"location: /typeProviders/carolyn-manager-cluster-py-type-apps->$.descriptorUrl\\\
    n\\\"{\\\\\\\"originalResponse\\\\\\\":\\\\\\\"{\\\\\\\\\\\\\\\"paths\\\\\\\\\\\
    \\\\\":[\\\\\\\\\\\\\\\"/apis\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/apis/\\\\\\\\\\\
    \\\\\",\\\\\\\\\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\\\\\\\\\",\\\\\\\\\\\\\
    \\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz\\\
    \\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz/etcd\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"\
    /healthz/log\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz/ping\\\\\\\\\\\\\\\",\-bash: Waiting: command not found

Could someone explain why if there is an error my cluster can still be setup? Or is there any special requirements that I need to set to avoid such error?

[mmosca]

I am seeing the same error as well.

{"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/mycluster-cluster-type->$.descriptorUrl\n\"{\\"originalResponse\\":\\"{\\\\"paths\\\\":[\\\\"/apis\\\\",\\\\"/apis/\\\\",\\\\"/apis/apiextensions.k8s.io\\\\",\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\",\\\\"/healthz\\\\",\\\\"/healthz/etcd\\\\",\\\\"/healthz/log\\\\",\\\\"/healthz/ping\\\\",\\\\"/healthz/poststarthook/crd-informer-synced\\\\",\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\",\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\",\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\",\\\\"/metrics\\\\",\\\\"/openapi/v2\\\\",\\\\"/version\\\\"]}\\",\\"reason\\":\\"The descriptor url 'https://SOME_IP/swaggerapi/api/v1' for type provider 'mycluster-cluster-type' could not be fetched.\\"}\""}

[ingernet]

I am also seeing this error, with a codebase that was successful when we ran it in November.

my-fancy-cluster-type: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/api/v1' for type provider 'my-fancy-cluster-type' could not be fetched.\\\"}\""}
my-fancy-cluster-type-apps: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type-apps->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/apis/apps/v1beta1' for type provider 'my-fancy-cluster-type-apps' could not be fetched.\\\"}\""}
my-fancy-cluster-type-v1beta1-extensions: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type-v1beta1-extensions->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/apis/extensions/v1beta1' for type provider 'my-fancy-cluster-type-v1beta1-extensions' could not be fetched.\\\"}\""}

[ANT1OFF]

I believe the reason is that GKE now defaults to version 1.14 where the swagger endpoint has been removed and replaced with the openapi/v2 endpoint. As a result, all custom types that use that endpoint fails. The error message is the default response from the API server when an unknown endpoint is accessed.

[mmosca]

The missing swagger endpoint can be fixed by replacing /swaggerapi/ with /openapi/v2/,

Line 97 in cluster.py should look like this: 'https://$(ref.', cluster_name, '.endpoint)/openapi/v2/',

I am not sure if that is enough to fix the issue, as I am getting some errors while trying to create namespaces on my modified script.

[rjg21]

Swapping the endpoint to /openapi/v2 removes the descriptor could not be fetched error but I now get this:

- code: MISSING_METHOD_IN_COLLECTION
   message: Method 'insert' does not exist for collection '/apis/batch/v1beta1/namespaces/{namespace}/cronjobs'
     in descriptor url 'https://<REDACTED>/openapi/v2'

[mmosca]

With some help from Google Engineers I managed to get my deployment working.

You no longer need multiple type providers, this should be enough:

resources.append({ 'name': type_name, 'type': 'deploymentmanager.v2beta.typeProvider', 'properties': { 'options': { 'validationOptions': {

Kubernetes API accepts ints, in fields they annotate

                # with string. This validation will show as warning
                # rather than failure for Deployment Manager.
                # https://github.com/kubernetes/kubernetes/issues/2971
                'schemaValidation': 'FAIL'
            },
            # According to kubernetes spec, the path parameter 'name' and 'namespace'
            # *must* be set inside the metadata field.
            # https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
            # The input mappings will map the values to path parameters
            'inputMappings': [{
                'fieldName': 'name',
                'location': 'PATH',
                'methodMatch': '^(get|delete|put|patch|post)$',
                'value': '$.resource.properties.metadata.name'
            }, {
                'fieldName': 'namespace',
                'location': 'PATH',
                'methodMatch': '^(get|delete|put|patch|post)$',
                'value': '$.resource.properties.metadata.namespace'
            }, {
                'fieldName': 'metadata.resourceVersion',
                'location': 'BODY',
                'methodMatch': '^(put|patch)$',
                'value': '$.resource.self.metadata.resourceVersion'
            }, {
                'fieldName': 'Authorization',
                'location': 'HEADER',
                'value': '$.concat("Bearer ",'
                         '$.googleOauth2AccessToken())'
            }]
        },
        'collectionOverrides': [{
                'collection': '/api/v1/namespaces/{namespace}/services/{name}',
                'options': {
                    'inputMappings': [{
                        'fieldName': 'spec.clusterIP',
                        'location': 'BODY',
                        'methodMatch': '^(put|patch)$',
                        'value': '$.resource.self.spec.clusterIP'
                    }]
                }
        }],
        'descriptorUrl': 'https://$(ref.' + cluster_name + '.endpoint)/openapi/v2'
    }
})

You have to remove extra qualifiers from your cluster_types and add {name} to the end of the url.

Something like this:

cluster_types = { 'Service': ''.join([ cluster_types_root, ':', '/api/v1/namespaces/{namespace}/services/{name}' ]), 'Deployment': ''.join([ cluster_types_root, ':', '/apis/apps/v1beta1/namespaces/{namespace}/deployments/{name}' ]), ...

[ingernet]

@mmosca, thanks for sharing your code! with a couple of tweaks, i got it working. here's my project fork if anybody needs a hand: https://github.com/ingernet/deploymentmanager-samples

[korutx]

@mmosca @ingernet thanks for sharing. I recently had the same problem, but the @ingernet code no longer works, because the Deployment Resource API changed once again. I will provide utility commands for diagnosis:

1. Get descriptor URL

   DESCRIPTOR_URL=`gcloud beta deployment-manager type-providers describe ${CLUSTER_NAME}-cluster-py-type --format='value(descriptorUrl)'`

2. Download the current swagger definition

   curl -sk -H "Authorization: Bearer `gcloud auth print-access-token`" $DESCRIPTOR_URL | jq > swagger_default.json 

3. Find the correct resource endpoint and adjust your references In my case this is the proper endpoint /apis/apps/v1/namespaces/{namespace}/deployments/{name}

I found this article really useful: https://medium.com/google-cloud/managing-kubernetes-customresourcedefinitions-with-google-deployment-manager-b9feeadae126

Hope it helps!