CustomResource field not set on the cluster as specified

[chirauki]

What happened?

We have some schemas that we obtain from running crd2pulumi. For one of them, we are seeing that spec is not being updated in the cluster.

The interface for the type looks like this:

/**
 * The set of arguments for constructing a Tier1Gateway resource.
 */
export interface Tier1GatewayArgs {
    apiVersion?: pulumi.Input<"install.tetrate.io/v1alpha1">;
    kind?: pulumi.Input<"Tier1Gateway">;
    metadata?: pulumi.Input<ObjectMeta>;
    /**
     * Specification of the desired state of the tsb management plane resource. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
     */
    spec?: pulumi.Input<{[key: string]: any}>;
}

Steps to reproduce

-

Expected Behavior

spec is propagated correctly to the destination object in k8s.

Actual Behavior

When creating the resource, I can see the preview --diff shows the right spec being computed:

[...]
    + kubernetes:install.tetrate.io/v1alpha1:Tier1Gateway: (create)
[...]
        apiVersion: "install.tetrate.io/v1alpha1"
        kind      : "Tier1Gateway"
        metadata  : {
            annotations: {
                foo                 : "bar"
                pulumi.com/autonamed: "true"
            }
            labels     : {
                app.kubernetes.io/managed-by: "pulumi"
            }
            name       : "tierone-tier1-2d00eaac"
            namespace  : "t1-92b9f5e6"
        }
        spec      : {
            service: {
                ports: [
                    [0]: {
                        name      : "http"
                        port      : 80
                        targetPort: 8080
                    }
                    [1]: {
                        name      : "mtls"
                        port      : 15443
                        targetPort: 15443
                    }
                ]
            }
        }

pulumi up --diff shows the same diff in the preview, but in the create, I can see:

    + kubernetes:install.tetrate.io/v1alpha1:Tier1Gateway: (create)
[...]
        apiVersion: "install.tetrate.io/v1alpha1"
        kind      : "Tier1Gateway"
        metadata  : {
            annotations: {
                foo                 : "bar"
                pulumi.com/autonamed: "true"
            }
            labels     : {
                app.kubernetes.io/managed-by: "pulumi"
            }
            name       : "tierone-tier1-028d9b1d"
            namespace  : "t1-92b9f5e6"
        }
        spec      : {
            service: {
                ports: [
                    [0]: {
                        name      : "http"
                        port      : 80
                        targetPort: 8080
                    }
                    [1]: {
                        name      : "mtls"
                        port      : 15443
                        targetPort: 15443
                    }
                ]
            }
        }

        --outputs:--
        metadata           : {
            annotations      : {
                foo                                             : "bar"
                kubectl.kubernetes.io/last-applied-configuration: (json) {
                    apiVersion: "install.tetrate.io/v1alpha1"
                    kind      : "Tier1Gateway"
                    metadata  : {
                        annotations: {
                            foo                 : "bar"
                            pulumi.com/autonamed: "true"
                        }
                        labels     : {
                            app.kubernetes.io/managed-by: "pulumi"
                        }
                        name       : "tierone-tier1-028d9b1d"
                        namespace  : "t1-92b9f5e6"
                    }
                    spec      : {
                        service: {
                            ports: [
                                [0]: {
                                    name      : "http"
                                    port      : 80
                                    targetPort: 8080
                                }
                                [1]: {
                                    name      : "mtls"
                                    port      : 15443
                                    targetPort: 15443
                                }
                            ]
                        }
                    }
                }

                pulumi.com/autonamed                            : "true"
            }
            creationTimestamp: "2022-11-14T18:27:11Z"
            generation       : 1
            labels           : {
                app.kubernetes.io/managed-by: "pulumi"
            }
            managedFields    : [
                [0]: {
                    apiVersion: "install.tetrate.io/v1alpha1"
                    fieldsType: "FieldsV1"
                    fieldsV1  : {
                        f:metadata: {
                            f:annotations: {
                            }
                            f:labels     : {
                            }
                        }
                        f:spec    : {
                            f:service: {
                            }
                        }
                    }
                    manager   : "pulumi-resource-kubernetes"
                    operation : "Update"
                    time      : "2022-11-14T18:27:11Z"
                }
            ]
            name             : "tierone-tier1-028d9b1d"
            namespace        : "t1-92b9f5e6"
            resourceVersion  : "22900756"
            uid              : "9d7216c0-efd8-4190-aa91-7a66e425ca1b"
        }

You can see how the outputs show the right info in the kubectl.kubernetes.io/last-applied-configuration annotation, and also includes the foo annotation, but the spec is not there. The same happens for create or update, the spec is ignored, but the kubectl.kubernetes.io/last-applied-configuration annotation is updated accordingly.

Output of pulumi about

No response

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[lblackstone]

Hi @chirauki -- as far as I can tell from those logs, this might be an issue with Server-Side Apply on your cluster. Since the last-applied-configuration annotation is present, I assume that you're not using SSA mode for your pulumi-kubernetes provider?

Assuming not, I suspect that another controller on the cluster may be managing the .spec.service.ports field. Notice that the managedFields output doesn't include the ports, which leads me to believe that it's not being managed by pulumi. You can check the full resource details by running the following command:

(I'm not 100% sure on the resource type. I put gateway, but that might not be correct)

# kubectl get <kind> <name> -n <namespace> -o json --show-managed-fields
kubectl get gateway tierone-tier1-028d9b1d -n t1-92b9f5e6 -o json --show-managed-fields

[chirauki]

NVM, this was a combination of lack of sleep and YAML engineering... We were missing a nested field, which caused the controller to (correctly) interpret the resource as empty, so it was applying the defaults. That's the same reason why updates did not work either.

Sorry to bother you... I will close this one.

[lblackstone]

Glad you figured it out :)