Manually changing CA certificate in AWS TLS inspection prevents further Pulumi updates

aaronnguyenwh commented 1 month ago

What happened?

When manually modifying the "CA certificate for outbound SSL/TLS inspection" in AWS TLS inspection configurations, Pulumi won't be able to make further changes to the CertificateAuthorityArn.

Example

const tlsInspectionConfiguration = new aws.networkfirewall.TlsInspectionConfiguration('egress', {
  name: 'egress',
  description: 'Configuration for outbound TLS inspection in Network Firewall',
  tlsInspectionConfiguration: {
    serverCertificateConfiguration: {
      certificateAuthorityArn: certArn,
      scopes: [
        {
          protocols: [6],
          destinationPorts: [
            {
              fromPort: 443,
              toPort: 443,
            },
          ],
          destinations: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
          sourcePorts: [
            {
              fromPort: 0,
              toPort: 65535,
            },
          ],
          sources: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
        },
      ],
    },
  },
});

Steps:

Set certificateAuthorityArn to certArn1 via Pulumi
Manually change the "CA certificate for outbound SSL/TLS inspection" in AWS TLS inspection configurations on AWS console to a different certArn2
Re-run Pulumi to update the certificateAuthorityArn to certArn1

Expected Behavior: The CA Arn is changed to certArn1 in both Pulumi and AWS Console

Actual Behavior: Pulumi returns error

     Type                                               Name                                                Status                  Info
     pulumi:pulumi:Stack                             ....us-gov-west-1                                    **failed**                1 error
 ~   └─ aws:networkfirewall:TlsInspectionConfiguration  egress                                          **updating failed**     [diff: ~tlsInspectionConfiguration];
                    "__meta": "{\"...\":{\"create\":300000000000,\"delete\
Diagnostics:
  pulumi:pulumi:Stack (....us-gov-west-1):
    error: update failed
                    "destinationCidrBlock": "...",
  aws:networkfirewall:TlsInspectionConfiguration (tls-egress):
    error: updating NetworkFirewall TLS Inspection Configuration (arn:aws-us-gov:network-firewall:us-gov-west-1:...:tls-configuration/egress): operation error Network Firewall: UpdateTLSInspectionConfiguration, https response error StatusCode: 400, RequestID: ..., InvalidTokenException: Update token isn't valid.
                    "instanceId": "",

Workaround: The only known solution is to remove the resource entirely and provision it again

Output of `pulumi about`


CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       darwin
Version  14.5
Arch     arm64```

### Additional context

_No response_

### Contributing

Vote on this issue by adding a 👍 reaction. 
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

justinvp commented 1 month ago

This looks like it may be specific to AWS. Moving to the aws repo for further triage.

corymhall commented 1 month ago

@aaronnguyenwh thanks for reporting this! When you run pulumi up to update the configuration are you running it with --refresh, or did you first run pulumi refresh? My hunch is that you first need to sync the cloud state through a refresh to get a valid update token.

aaronnguyenwh commented 1 month ago

@corymhall Yes, I did. But pulumi refresh output for certificateAuthorityArn doesn't match the actual one on TLS inspection configuration.

corymhall commented 1 month ago

@aaronnguyenwh do you have an example of creating the certificate? I'm trying, but can't figure out how to create one that networkfirewall accepts.

aaronnguyenwh commented 1 month ago

@corymhall our usage is very similar to this snippet:

import * as aws from "@pulumi/aws";
import * as tls from "@pulumi/tls";

const rootCA = new aws.acmpca.CertificateAuthority("root-ca", {
    type: "ROOT",
    certificateAuthorityConfiguration: {
        keyAlgorithm: "RSA_4096",
        signingAlgorithm: "SHA512WITHRSA",
        subject: {
            commonName: "Root CA",
        },
    },
    permanentDeletionTimeInDays: 7,
});

const key = new tls.PrivateKey("key", {algorithm: "RSA"});
const csr = new tls.CertRequest('csr', {
    privateKeyPem: key.privateKeyPem,
    subject: {
        commonName: 'Network Firewall',
    },
});

const tlsCert = new aws.acm.Certificate("tls-cert", {
    certificateAuthorityArn: rootCA.arn,
    privateKey: key.privateKeyPem,
    certificateSigningRequest: csr.certRequestPem,
});

// then you can use `tlsCert` for TLS inspection configuration

More insights on some of the restrictions TLS inspection has CA certificate - Outbound SSL/TLS inspection

To configure outbound TLS inspection, you must first import a certificate authority (CA) certificate into AWS Certificate Manager (ACM). After you import the CA certificate in ACM, you can associate the CA certificate with your TLS inspection configuration. Network Firewall uses the CA certificate to generate a server certificate, which the service uses to establish trust between the client and the server.

corymhall commented 1 month ago

Unfortunately I'm unable to get anything working. I get an error from networkfirewall. It would help to have a self contained example that we can deploy to reproduce, otherwise I can try to get something working as time permits.

CertificateAuthorityArn is invalid because it references a certificate authority that doesn't comply with RFC 5280 basic constraints.

aaronnguyenwh commented 1 month ago

@corymhall can you try doing this manually with openssl?

Create a file named subordinateCA.cnf with the following contents:

[ v3_ca ]
basicConstraints = critical,CA:TRUE,pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer

Run the following commands to create the certificate authorities using OpenSSL:

openssl genpkey -algorithm RSA -out rootCA.key -pass pass:password -pkeyopt rsa_keygen_bits:4096
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt -passin pass:password -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=RootCA"

openssl genpkey -algorithm RSA -out subordinateCA-1.key -pkeyopt rsa_keygen_bits:4096
openssl req -new -key subordinateCA-1.key -out subordinateCA-1.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=SubordinateCA"
openssl x509 -req -in subordinateCA-1.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out subordinateCA-1.crt -days 3650 -sha256 -extfile subordinateCA.cnf -extensions v3_ca -passin pass:password
cat rootCA.crt subordinateCA-1.crt > subordinateCAChain-1.pem

openssl genpkey -algorithm RSA -out subordinateCA-2.key -pkeyopt rsa_keygen_bits:4096
openssl req -new -key subordinateCA-2.key -out subordinateCA-2.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=SubordinateCA"
openssl x509 -req -in subordinateCA-2.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out subordinateCA-2.crt -days 3650 -sha256 -extfile subordinateCA.cnf -extensions v3_ca -passin pass:password
cat rootCA.crt subordinateCA-2.crt > subordinateCAChain-2.pem

Upload the subordinateCA-1.crt and subordinateCA-2.crt certificate to ACM along with their respective chain and key files.
Create a tls inspection configuration through Pulumi that uses the subordinateCA-1 ARN as the CA certificate.
In the AWS console, go to manually change the CA certificate and notice that the dropdown is not set when you go to change it, similar to screenshot below
Manually set the certificate to the subordinateCA-1 certificate in the dropdown
Run a pulumi refresh and then change the certificateAuthorityArn to subordinateCA-2 ARN in the Pulumi code
Run pulumi up and see the error

mhamill2 commented 1 month ago

It seems like the update token isn't being updated. When you do a refresh after manually changing the tls certificate in the console, the update token does not change when looking at the stack export for that tls inspection resource.

aaronnguyenwh commented 3 weeks ago

@corymhall can we try this?

import * as aws from '@pulumi/aws';
import * as tls from '@pulumi/tls';

const privateKey1 = new tls.PrivateKey('privateKey-1', { algorithm: 'RSA' });
const selfSignedCert1 = new tls.SelfSignedCert('selfSignedCert-1', {
  allowedUses: ['digital_signature', 'cert_signing', 'crl_signing'],
  privateKeyPem: privateKey1.privateKeyPem,
  validityPeriodHours: 9000,
  dnsNames: ['firewall.test.com'],
  isCaCertificate: true,
  setSubjectKeyId: true,
  setAuthorityKeyId: true,
  subject: {
    commonName: 'test',
    country: 'us',
    organization: 'testing',
  },
});

const privateKey2 = new tls.PrivateKey('privateKey-2', { algorithm: 'RSA' });
const selfSignedCert2 = new tls.SelfSignedCert('selfSignedCert-2', {
  allowedUses: ['digital_signature', 'cert_signing', 'crl_signing'],
  privateKeyPem: privateKey2.privateKeyPem,
  validityPeriodHours: 9000,
  dnsNames: ['firewall.test.com'],
  isCaCertificate: true,
  setSubjectKeyId: true,
  setAuthorityKeyId: true,
  subject: {
    commonName: 'test',
    country: 'us',
    organization: 'testing',
  },
});

const acmCertificate1 = new aws.acm.Certificate(
  'acmCertificate-1',
  {
    privateKey: selfSignedCert1.privateKeyPem,
    certificateBody: selfSignedCert1.certPem,
    certificateChain: selfSignedCert1.certPem,
  },
  { replaceOnChanges: ['certificateBody'] },
);
const acmCertificate2 = new aws.acm.Certificate(
  'acmCertificate-2',
  {
    privateKey: selfSignedCert2.privateKeyPem,
    certificateBody: selfSignedCert2.certPem,
    certificateChain: selfSignedCert2.certPem,
  },
  { replaceOnChanges: ['certificateBody'] },
);

new aws.networkfirewall.TlsInspectionConfiguration('tlsInspectionConfig', {
  description: 'Configuration for outbound TLS inspection in Network Firewall',
  tlsInspectionConfiguration: {
    serverCertificateConfiguration: {
      // certificateAuthorityArn: acmCertificate1.arn,
      certificateAuthorityArn: acmCertificate2.arn,
      checkCertificateRevocationStatus: {
        revokedStatusAction: 'REJECT',
        unknownStatusAction: 'REJECT',
      },
      scopes: [
        {
          protocols: [6],
          destinationPorts: [
            {
              fromPort: 443,
              toPort: 443,
            },
          ],
          destinations: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
          sourcePorts: [
            {
              fromPort: 0,
              toPort: 65535,
            },
          ],
          sources: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
        },
      ],
    },
  },
});

corymhall commented 3 weeks ago

@aaronnguyenwh thanks for the example, I was able to reproduce the issue! It looks like it is a bug in the upstream terraform provider where the UpdateToken is not being read as part of refreshing the resource data. https://github.com/hashicorp/terraform-provider-aws/issues/38487

pulumi / pulumi-aws