Pulumi preview and up incorrectly says that it will change / is changing a service

JiriKovar commented 1 week ago

Describe what happened

Hello,

In case of some of our service, the pulumi preview and pulumi up commands shows that it will do "something" with a service (the service root is flagged with ~, prints details of the service, but in the details, nothing is flagged to be changed (no line starts with ~/-/+).

In case of pulumi up, it says that its updating the services, the service is listed in the number of the updated services, but at the end of the day, it does nothing at all = not even a new version of the fastly service is created, which shows that at the runtime the Pulumi detects that there is no change - version management requires extra requests AFAIK and its not running even these requests. Even so, the next round of pulumi preview and pulumi up shows the same results.

Have you ever observed the same behaviour? Do you have any idea what could be the cause? In case of some of the types of our Fastly services, it happens only in certain configuration (for QA configurations some objects like ACLs or WAF are missing), in some of them it doesn't happen (even though they have ACLs and WAF), in some of them it happens consistently no matter the configuration.

This has been bothering us for quite a while, we had a couple of theories, but none of them proved to be right - it could be some handling of whitespaces, or whatnot. I'm reporting it as a bug, since the logs and the actual behaviour does not match, but at the same time I'm asking for assistance, because I cannot pinpoint the cause and give you a reasonable repro.

Sample program

import { ServiceVcl } from '@pulumi/fastly';
import type {
  ServiceVclBackend,
  ServiceVclCacheSetting,
  ServiceVclDictionary,
  ServiceVclDomain,
  ServiceVclHeader,
} from '@pulumi/fastly/types/input';
import * as pulumi from '@pulumi/pulumi';
import { WebTest, type WebTestArgs } from '../../../@components/azure/webTest';
import type { FastlyMonitoringConfig } from '../../../@components/fastly/fastlyMonitoringConfig';
import type { Domains, ServiceBackend } from '../../../@components/fastly/types';
import { Waf, type WafConfig } from '../../../@components/fastly/waf/waf';
import type { ResourceContext } from '../../../@components/utils/resourceContext';

export interface FastlyServiceWhereItAlwaysHappensArgs {
  context: ResourceContext;
  domains: Domains;
  monitoring: FastlyMonitoringConfig | undefined;
  serviceBackend: ServiceBackend;
  waf: WafConfig | undefined;
}

export class FastlyServiceWhereItAlwaysHappens {
  public readonly id: pulumi.Output<string>;
  public readonly name: pulumi.Output<string>;
  public readonly primaryDomain: pulumi.Output<string>;

  constructor(name: string, args: FastlyServiceWhereItAlwaysHappensArgs) {
    const domains: ServiceVclDomain[] = [args.domains.primary, ...args.domains.other].map((domain) => ({
      name: domain,
    }));
    const primaryDomain = args.domains.primary;

    const cacheSettings: ServiceVclCacheSetting[] = [
      {
        name: 'Do not cache',
        action: 'pass',
        ttl: 0,
        staleTtl: 0,
      },
    ];

    const backends: ServiceVclBackend[] = [
      {
        address: args.serviceBackend.address,
        name: `Backend_${args.serviceBackend.name.replace(/[^a-zA-Z0-9]+/, '_')}`,
        overrideHost: args.serviceBackend.address,
        port: 443,
        useSsl: true,
        sslCertHostname: '*.azurewebsites.net',
        autoLoadbalance: false,
        betweenBytesTimeout: 20000,
        connectTimeout: 20000,
        firstByteTimeout: 300000,
      },
    ];

    const headers: ServiceVclHeader[] = [
      {
        name: 'X-Azure-FDID',
        action: 'set',
        type: 'request',
        destination: 'http.X-Azure-FDID',
        source: pulumi.interpolate`\"${args.serviceBackend.azureFrontDoorId}\"`,
        priority: 10,
      },
    ];

    const dictionaries: ServiceVclDictionary[] = [
      {
        name: 'Edge_Security',
      },
    ];

    const serviceVcl = new ServiceVcl(
      name,
      {
        backends,
        cacheSettings,
        defaultTtl: 3600,
        dictionaries,
        domains,
        headers,
        dynamicsnippets: [],
      },
      {
        ignoreChanges: ['versionComment', 'dynamicsnippets'],
      },
    );

    if (args.waf) {
      new Waf(
        `${name}-waf`,
        {
          ...args.waf,
          origins: backends.map((b) => b.address),
          serviceId: serviceVcl.id,
        },
        {
          parent: serviceVcl,
        },
      );
    }

    if (args.monitoring) {
      for (const domain of [primaryDomain, ...args.domains.other]) {
        new WebTest(
          `subscription-cdn-webtest-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
          {
            resourceGroup: args.monitoring.resourceGroup,
            webTestProperties: {
              name: `subscription-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
              url: `https://${domain}/`,
              expectedHttpStatusCode: 200,
            },
            azureSubscriptionId: args.monitoring.azureSubscriptionId,
            applicationInsights: args.monitoring.applicationInsights,
            alert: {
              description: '      Website is DOWN',
              monitoringEmails: [args.monitoring.emails.service],
              actionGroups: [args.monitoring.actionGroupId.azureAlertMessageFormatter],
              severityLevel: 3,
            },
          } as WebTestArgs,
          {
            parent: serviceVcl,
          },
        );
      }
    }

    this.id = serviceVcl.id;
    this.name = serviceVcl.name;
    this.primaryDomain = pulumi.output(primaryDomain);
  }
}

Log output

@ updating....
    ~ fastly:index/serviceVcl:ServiceVcl: (update)
        [id=<REDACTED_SERVICE_ID>]
        [urn=urn:pulumi:qa::persistent-fastly::fastly:index/serviceVcl:ServiceVcl::qa-serviceWhereItAlwaysHappens]
        [provider=urn:pulumi:qa::persistent-fastly::pulumi:providers:fastly::default_8_11_0::021b2e85-0f62-46e3-9bea-a3381e86b531]
        activate       : true
        backends       : [secret]
        cacheSettings  : [
            [0]: {
                action        : "pass"
                cacheCondition: ""
                name          : "Do not cache"
                staleTtl      : 0
                ttl           : 0
            }
        ]
        comment        : "Managed by Terraform"
        defaultTtl     : 3600
        dictionaries   : [
            [0]: {
                forceDestroy: false
                name        : "Edge_Security"
                writeOnly   : false
            }
        ]
        domains        : [
            [0]: {
                name      : "serviceWhereItAlwaysHappens.ourinternaldomain.com"
            }
            [1]: {
                name      : "qa-serviceWhereItAlwaysHappens.global.ssl.fastly.net"
            }
        ]
        dynamicsnippets: []
        headers        : [
            [0]: {
                action           : "set"
                cacheCondition   : ""
                destination      : "http.X-Azure-FDID"
                ignoreIfSet      : false
                name             : "X-Azure-FDID"
                priority         : 10
                requestCondition : ""
                responseCondition: ""
                source           : "\"<REDACTED_GUID>\""
                type             : "request"
            }
        ]
        http3          : false
        name           : "qa-serviceWhereItAlwaysHappens"
        staleIfError   : false
        staleIfErrorTtl: 43200
Resources:
    ~ 1 updated
    116 unchanged

Affected Resource(s)

No response

Output of `pulumi about`

CLI          
Version      3.136.1
Go Version   go1.23.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       Microsoft Windows Server 2022 Standard
Version  10.0.20348 Build 20348
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v22.9.0'

Backend        
Name           Agent-06
URL            azblob://qastacks?storage_account=***
User           Agent-06\Administrator
Organizations  
Token type     personal

Pulumi locates its logs in C:\Users\ADMINI~1\AppData\Local\Temp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

JiriKovar commented 1 week ago

FYI I have also tried to run pulumi refresh --diff --skip-preview --non-interactive -j and analyzed the JSON output, where there are no differences between the "new" and the "old" objects in the results are the same and the "detailedDiff" is null.

guineveresaenger commented 6 days ago

Hi @JiriKovar, thank you for reporting this. We do have a category of bugs involving spurious diffs, and we understand they're confusing an unwieldy for our users.

Some of these bugs stem from the pulumi-terraform bridge rather than an individual provider. In fact, we're currently working on improved diffing logic in the bridge. Other bugs of this type are rooted in the upstream provider implementation, such as this one on ServiceVcl bigquery logging.

Even though they may have different causes, we'd love it if you could send us a sample program with such an unexplained diff.

pulumi / pulumi-fastly