Detecting a diff in GCP pubsub logging config

[alex-deng-featurehub]

What happened?

We are using GCP pubsub logging (https://www.pulumi.com/registry/packages/fastly/api-docs/servicevcl/#servicevcllogginggooglepubsub) in our Fastly VCL service and we are getting a diff when we run pulumi up despite having made no change.

Steps to reproduce

We have the below block in our VCL object

loggingGooglepubsubs: [{
  name: 'fastly-logging',
  projectId: gcpConfig.gcpProjectName,
  secretKey: jsonObject['private_key'],
  topic: args.fastlyLogsTopicName,
  user: gcpConfig.fastlyServiceAccountEmail
}]

Expected Behavior

We have added a ignoreChanges: ["loggingGooglepubsubs[0]"] block to ServiceVcl as we do not expect it to have any diff.

Actual Behavior

Values contain sensitive data and have been hidden but there is no diff

    ~ fastly:index/serviceVcl:ServiceVcl: (update)
        [id=HIDDEN]
        [urn=urn:pulumi:test::local_saas::fastly:index/serviceVcl:ServiceVcl::HIDDEN]
        [provider=urn:pulumi:test::local_saas::pulumi:providers:fastly::default_4_0_4::e66d2abb-e2a4-480c-af5e-492dbee86e51]
      ~ loggingGooglepubsubs: [
          ~ [0]: {
                  ~ formatVersion: 2 => 2
                  ~ name         : "fastly-logging" => "fastly-logging"
                  ~ projectId    : "HIDDEN" => "HIDDEN"
                  ~ topic        : "HIDDEN" => "HIDDEN"
                  ~ user         : "HIDDEN" => "HIDDEN"
                }
        ]

Versions used

4.0.4

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[earamendi-globant]

Hi there.

Same behaviour when using fastly v5.0.0

[AaronFriel]

@alex-deng-featurehub @earamendi-globant Hey folks, could either of you contribute a minimal Pulumi program in any language to help us reproduce this, or let us know if we might have already fixed this behavior?

[rvowles]

I tried upgrading this with @alex-deng-featurehub to v6 by the

kubernetes:core/v1:Namespace (google-managed-prometheus):
    error: failed to initialize discovery client: The gcp auth plugin has been removed.
    Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
    See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details

issue is blocking us, there appears to be no documentation on how to tell pulumi to use the new config or whatever.

[AaronFriel]

Hey @rvowles, thanks for replying. I'm not sure if I can reproduce this from that error message. A minimal program would be really helpful.

That said, to troubleshoot your issue with GKE auth, there are several ways

If you're using ambient default credentials

Ensure that Pulumi runs with the correct kubectl context set. In an interactive shell, you'd typically run something like:

gcloud container clusters get-credentials mygkecluster

If that command succeeds, for your cluster name, it should populate your Kubernetes config, usually located at ~/.kube/config.

If you're using a stack configuration credentials

Update the kubernetes:kubeconfig property in your stack with:

pulumi config set --secret kubernetes:kubeconfig "$(cat PATH_TO_KUBECONFIG)"

If you're using explicit provider credentials

Ensure that you're passing in a valid kubeconfig field in the constructor:

const provider = new kubernetes.Provider("k8s-provider", {
  kubeconfig: ...
});

[mikhailshilkov]

Since we haven't heard back, I'll go ahead and close this issue as stale. Please open a new issue if the problem still persists and provide the details for us to reproduce it. Thank you!

[rvowles]

We just upgraded to "@pulumi/fastly": "^8.1.3", and it still isn't fixed, I am trying to delve deeper into it. It is happy with everything but the logging!

[rvowles]

So it keeps saying the format has changed (from 2 => 2) and the secret has changed (it hasn't)

 ~ loggingGooglepubsubs: [
          ~ [0]: {
                  ~ format           : 
                        ...
                          "response_status": %{resp.status}V
                        }
                  ~ formatVersion    : 2 => 2
                  ~ name             : "featurehub-fastly-logging" => "featurehub-fastly-logging"
                  ~ projectId        : "featurehub-saas-non-prod" => "featurehub-saas-non-prod"
                  ~ responseCondition: "only-log-when" => "only-log-when"
                  ~ secretKey        : [secret] => "-----BEGIN PRIVATE KEY-----<preview-dump-of-whole-private-key here>\n-----END PRIVATE KEY-----\n"
                  ~ topic            : "test-fastly-logs-topic-f7e6357" => "test-fastly-logs-topic-f7e6357"
                  ~ user             : "fastly-service-account@blahblah" => "fastly-service-account@blahblah"
                }
        ]

[mikhailshilkov]

@rvowles Can you please share a minimal repro program with us?

[derence-boxed]

Appreciate if we can reopen this issue, I am using provider version 8.6.0.

Apparently Pulumi saved this input as secret in the state file and the keys aren't the same as the code, therefore we can't use ignore loggingGooglepubsubs[0].secretKey to prevent Pulumi from redeploying the Fastly service as I guess secretKey is something dynamic (referred from the output of the service account key) and hence Pulumi thinks it might be a new value and forcing an update. "loggingGooglepubsubs": { "4dabf18193072939515e22a98388d": "1b47061264138c30d75fd1eb44270", "ciphertext": "[secret]" },

Is it possible to change the resource record in the state file like how Terraform do it? "logging_googlepubsub": [ { "format": "fcfe8", "format_version": 2, "name": "9fccd", "placement": "78855", "project_id": "887d7", "response_condition": "78855", "secret_key": "78480", "topic": "1d8e5", "user": "54cd7" } ],

[mikhailshilkov]

@derence-boxed Could you please share a repro code?

Is it possible to change the resource record in the state file like how Terraform do it?

Yes, you should be able to export your state file with pulumi state export, edit it, and then pulumi state import.

[derence-boxed]

Create the service account and the key, and extract the private key JSON file and this is parsed to fastly:

    const serviceAccount = new gcp.serviceaccount.Account(`${name}-service-account`, {
        accountId: `${props.name}`,
        displayName: `${props.name}`,
        project: props.gcpProjectName,
    }, {
        parent: this,
    });
        const serviceAccountGCPCredsFile = new gcp.serviceaccount.Key(`${name}-service-acccount-gcp-creds-file`, {
            serviceAccountId: serviceAccount.id,
            privateKeyType: 'TYPE_GOOGLE_CREDENTIALS_FILE',
        });
        this.serviceAccountPrivateKeyJsonFile = pulumi.all([serviceAccountGCPCredsFile.privateKey]).apply(([privateKey]) => Buffer.from(privateKey,'base64').toString());
        this.serviceAccountPrivateKeyOnlyJsonFile = pulumi.all([this.serviceAccountPrivateKeyJsonFile]).apply(([serviceAccountPrivateKeyJsonFile]) => JSON.parse(serviceAccountPrivateKeyJsonFile).private_key);

---

Fastly service:

    const fastlyService = new fastly.ServiceVcl(`${name}`, {
        name: props.name,
        backends: props.backendConfig,
        domains: [
            {
                comment: props.domainComment,
                name: props.domainName,
            },
        ],
        headers: [
            {
                action: 'set',
                destination: 'http.Strict-Transport-Security',
                ignoreIfSet: false,
                name: 'Force TLS and enable HSTS',
                priority: 100,
                source: '\"max-age=31536000\"',
                type: 'response',
            },
        ],
        gzips: [
            {
                name: 'file extensions and content types',
                extensions: ['css', 'js', 'html', 'eot', 'ico', 'otf', 'tff', 'json', 'svg'],
                contentTypes: [
                    'text/html',
                    'application/x-javascript',
                    'text/css',
                    'application/javascript',
                    'text/javascript',
                    'application/json',
                    'application/xml',
                    'font/eot',
                    'font/opentype',
                    'font/otf',
                    'image/svg+xml',
                    'text/plain',
                    'text/xml',
                ],
            },
        ],
        conditions: props.conditions,
        snippets: props.snippets,
        requestSettings: [
            {
                name: 'Force TLS and enable HSTS',
                bypassBusyWait: false,
                forceSsl: true,
                forceMiss: false,
                geoHeaders: false,
                maxStaleAge: 0,
                xff: 'append',
            },
        ],
        vcls: props.vcls,
        forceDestroy: true,
        loggingGooglepubsubs: [{
            name: 'datadog-pubsub',
            projectId: props.gcpProjectName,
            topic: fastlyDatadogRedirectTopic.name,
            format: fs.readFileSync(`${__dirname}/datadome/loggingFormat.vcl`, 'utf-8'),
            user: fastlyPubsubServiceAccount.serviceAccountEmail,
            secretKey: fastlyPubsubServiceAccount.serviceAccountPrivateKeyOnlyJsonFile,
        }],
    }, {
        parent: this,
        dependsOn: [
            fastlyDatadogRedirectTopic,
            fastlyDatadogRedirectDLQTopic,
            fastlyDatadogRedirectSubscription,
            fastlyPubsubServiceAccount,
        ],
    });

---

My guess is because fastlyPubsubServiceAccount.serviceAccountPrivateKeyOnlyJsonFile is a promise in the service account class of which Pulumi might think the value could be different, as such loggingGooglepubsubs is marked to be updated.

Ideally, since there is no change in the private key, there is no need to update the whole fastly service due to the secret key "change" in loggingGooglepubsubs, otherwise, is it possible to change how Pulumi write the state file for loggingGooglepubsubs like that in Terraform, such that we can ignore changes only on the secretKey?

[quote]Yes, you should be able to export your state file with pulumi state export, edit it, and then pulumi state import.[\quote] I mean it is how Pulumi record loggingGooglepubsubs in the state file, instead of editing the state file manually and import it.

[iwahbe]

@derence-boxed Thanks for posting a repro. We have added this issue to our diff bug queue.

[VenelinMartinov]

I've attempted to repro this with various Fastly and GCP versions and variations on the following program:

import * as pulumi from "@pulumi/pulumi";
import * as fastly from "@pulumi/fastly";
import * as gcp from "@pulumi/gcp";
import * as random from "random";

const pubsubTopic = new gcp.pubsub.Topic("my-topic", {
  name: "my-fastly-logs-topic",
});

const pet = new random.RandomPet("my-pet", {});

const serviceAccount = new gcp.serviceaccount.Account("my-service-account", {
  accountId: pet.id,
  displayName: "My Service Account",
});

const serviceAccountKey = new gcp.serviceaccount.Key("my-service-key", {
  serviceAccountId: serviceAccount.name,
  privateKeyType: "TYPE_GOOGLE_CREDENTIALS_FILE",
});

const private_decoded = serviceAccountKey.privateKey
  .apply((key) => Buffer.from(key, "base64").toString("ascii"))
  .apply((key) => JSON.parse(key).private_key);

new fastly.ServiceVcl("my-service", {
  forceDestroy: true,
  domains: [
    {
      name: "fastly.pulumi.com",
    },
  ],
  backends: [
    {
      address: "fastly.pulumi.com",
      name: "example-backend",
    },
  ],

  loggingGooglepubsubs: [
    {
      name: "fastly-logging",
      projectId: serviceAccount.accountId,
      secretKey: private_decoded,
      topic: pubsubTopic.name,
      user: serviceAccount.email,
    },
  ],
});

The resource provisions fine and I can preview it without getting any diffs afterwards.

Whoever is still affected by this, can you please try upgrading fastly to v8.12.0 and GCP to https://github.com/pulumi/pulumi-gcp/releases/tag/v8.4.1 and retrying? We have had some significant improvements in the area, so this might be fixed.

If you are still hitting this can you please record some debug logs by prefixing the pulumi preview command with PULUMI_DEBUG_GRPC="grpc.json" like so:

PULUMI_DEBUG_GRPC="grpc.json" pulumi preview

and posting the GRPC logs here, making sure to delete any sensitive data.

[VenelinMartinov]

Okay, it was pointed out to me that the issue is very old and some of the original reporters were using fastly 5.0.0. With that piece of information I did successfully repro the problem.

After upgrading to the newest version of the Fastly provider, the resource shows a diff during the first up:

      + http3               : false
      ~ loggingGooglepubsubs: [
          ~ [0]: {
                  - format       : [secret]
                  ~ formatVersion: [secret] => [secret]
                  ~ name         : [secret] => [secret]
                  ~ projectId    : [secret] => [secret]
                  ~ secretKey    : [secret] => [secret]
                  ~ topic        : [secret] => [secret]
                  ~ user         : [secret] => [secret]
                }
        ]
Resources:
    ~ 1 to update
    5 unchanged

But shows no further diffs on successive previews without any changes to the resourcce.

I am going to close this as fixed, please, let me know if anyone is still having issues here on the newest fastly version.