pulumi / pulumi-policy-aws

A policy pack of rules to enforce AWS best practices for security, reliability, cost, and more!
https://www.pulumi.com
Apache License 2.0
33 stars 6 forks source link

acm-certificate-expiration check EHOSTDOWN error preventing builds #95

Closed kwamae-webflow closed 1 year ago

kwamae-webflow commented 1 year ago

What happened?

We are attempting to build our projects, but we're receiving an error relating to the acm-certificate-expiration check that is preventing that. We are experiencing this across multiple machines. This is the acm-certificate-expiration check in pulumi-awsguard.

Steps to reproduce

We've removed and disabled the ACM check, and our builds pass but fail when we re-able it.

Expected Behavior

The ACM check should pass and allow the Pulumi project to build

Actual Behavior

'Pulumi Up' reports the ACM error and fails the build.

Output of pulumi about

0110 14:33:04.612352   15509 rpc.go:74] Marshaling property for RPC[]: version={5.14.0}
I0110 14:33:09.215482   15509 analyzer_plugin.go:265] Analyzer[pulumi-awsguard].AnalyzeStack(...) failed: err=Error validating stack with policy acm-certificate-expiration:
Error: connect EHOSTUNREACH 169.254.169.254:80 - Local (192.168.2.3:54206)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at net.js:1022:9
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
I0110 14:33:09.215557   15509 deployment_executor.go:327] deploymentExecutor.Execute(...): error analyzing resources: Error validating stack with policy acm-certificate-expiration:
Error: connect EHOSTUNREACH 169.254.169.254:80 - Local (192.168.2.3:54206)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at net.js:1022:9
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
I0110 14:33:09.215583   15509 eventsink.go:86] eventSink::Error(<{%reset%}>Error validating stack with policy acm-certificate-expiration:
Error: connect EHOSTUNREACH 169.254.169.254:80 - Local (192.168.2.3:54206)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at net.js:1022:9
    at processTicksAndRejections (internal/process/task_queues.js:77:11)<{%reset%}>)
I0110 14:33:09.215641   15509 deployment_executor.go:138] deploymentExecutor.Execute(...): exiting provider canceller
I0110 14:33:09.269624   15509 ignore.go:44] Explicitly ignoring and discarding error: rpc error: code = Canceled desc = grpc: the client connection is closing
I0110 14:33:09.269867   15509 host.go:530] Error closing 'nodejs' language plugin during shutdown; ignoring: 1 error occurred:
        * operation not permitted

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

RobbieMcKinstry commented 1 year ago

Hello Kwamae, thanks for opening this issue. It appears to be specific to Pulumi AWSGuard, so I'm going to transfer this issue to https://github.com/pulumi/pulumi-policy-aws/issues

danielrbradley commented 1 year ago

Hi @kwamae-webflow could I just check a couple of details here:

  1. You've attached the stack trace within the pulumi about output - is this the command that's causing the issue or is it actually when doing an operation such as pulumi up?
  2. Please could you provide a minimal program which reproduces the issue (either inline or a link to a gist or similar)?
  3. Please could you include environment information (OS version, Pulumi CLI version, CPU arch, plugin versions).

Thanks

kwamae-webflow commented 1 year ago

Hello @danielrbradley, I've put my responses below.

  1. Pulumi Up is the command causing the issue.

  2. Puluimi Program

import * as pulumi from '@pulumi/pulumi';

// eslint-disable-next-line workspaces/require-dependency
import {AcmCert} from "@modules/acm-cert";

import {IamServiceAccount} from "./resources/aws";
import {AppComponent} from "./resources/k8s";
import * as aws from "@pulumi/aws";

type Output = {
    acmCert: AcmCert;
    iamServiceAccount: IamServiceAccount;
    appComponent: AppComponent;
}

export = async (): Promise<Output> => {
    const stack = pulumi.getStack();

    const tags = {
        env: stack,
        environment: stack,
        createdBy: 'pulumi',
        pulumiProject: pulumi.getProject(),
        service: 'ses',
        owner: stack,
    };

    // StageHand needs namespace to be "default"
    const resourceName = 'ep-ses-utils';
    const k8sNamespace = 'default';
    const k8sServiceAccountName = `${resourceName}-sa`;

    const awsIdentity = await aws.getCallerIdentity();
    const webflowBatchStack = new pulumi.StackReference(`webflow/webflow-batch/${stack}`);
    const logicBatch = await webflowBatchStack.getOutputValue('logic');
    const logicQueueConfig = logicBatch.queue;

    const config = new pulumi.Config();
    const clusterName = config.require("eksClusterName");
    const clusterOIDCProvider = config.require("eksClusterOIDCProvider");
    const queueArn = config.require("sqsQueueArn");
    const redisHost = config.require("redisPrimaryEndpoint");
    const sqsUrl = config.require("sqsUrl");
    const sqsLowPriorityUrl = config.require("sqsLowPriorityUrl");
    const lambdaName = config.require("lambdaName");
    const lambdaArn = config.require("lambdaArn");

    // TODO: SES-Utils uses split DNS for domain "kube.webflow.services".
    //  Given that Webflow don't have a PKI ready for Private CA,
    //  we need to do TLS DNS challenge in the Route53 Public Hosted Zone,
    //  but only intended to be used for Route53 Private Hosted Zone.

    //  TODO: Once a PKI is created by Reliability/Security,
    //   then we can generate internal certificates for Private Hosted Zones.
    //   See comments in Jira ticket LOGIC-3889 for more info of this tech debt.
    const acmCert = new AcmCert('acm-cert', {
        tags: tags,
        domains: [
            {
                name: `${resourceName}.${clusterName}.kube.webflow.services`,
                hostedZoneId: config.require('externalDnsZone')
            }
        ]
    });

    const iamServiceAccount = new IamServiceAccount("aws-component", {
        tags,
        resourceName,
        clusterName,
        clusterOIDCProvider,
        queueArn,
        logicQueueArn: logicQueueConfig.arn,
        k8sNamespace,
        k8sServiceAccountName,
        accountId: awsIdentity.accountId,
        lambdaArn,
    });

    const appComponent = new AppComponent("k8s-component", {
        tags,
        resourceName,
        clusterName,
        redisHost,
        sqsUrl,
        sqsLowPriorityUrl,
        logicQueueName: logicQueueConfig.name,
        logicQueueUrl: logicQueueConfig.url,
        lambdaName,
        role: iamServiceAccount.role,
        acmCert: acmCert,
        k8sNamespace,
        k8sServiceAccountName,
        minReplicas: config.requireNumber('minReplicas'),
        maxReplicas: config.requireNumber('maxReplicas')
    });

    return {
        acmCert,
        iamServiceAccount,
        appComponent
    };
}
  1. Pulumi About
CLI          
Version      3.51.0
Go Version   go1.19.4
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host     
OS       darwin
Version  13.1
Arch     arm64

This project is written in nodejs: executable='/usr/local/bin/node' version='v18.12.1'

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/kwamae-mcpherson-webflow-com
User           kwamae-mcpherson-webflow-com
Organizations  kwamae-mcpherson-webflow-com, webflow

Pulumi locates its logs in /var/folders/qb/gz6gcmjd57185msmv8kj4ngm0000gp/T/ by default
warning: Failed to get information about the Pulumi program's dependencies: could not find either /Users/kwamaemcpherson/Library/Mobile Documents/com~apple~CloudDocs/Repos/infrastructure/pulumi/projects/ses-utils/yarn.lock or /Users/kwamaemcpherson/Library/Mobile Documents/com~apple~CloudDocs/Repos/infrastructure/pulumi/projects/ses-utils/package-lock.json
warning: Failed to get information about the current stack: No current stack
warning: A new version of Pulumi is available. To upgrade from version '3.51.0' to '3.51.1', run 
   $ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
thomas11 commented 1 year ago

Hi @kwamae-webflow, sorry for the delay here.

The error EHOSTUNREACH doesn't necessarily suggest a Pulumi issue. Could it be a local connectivity problem? For instance, another user recently fixed his connection timeout by disabling Little Snitch.

In case that doesn't help, would you be able to attach a full debug output via pulumi up --logtostderr --logflow -v=9 2> out.txt?

kwamae-webflow commented 1 year ago

@thomas11 Hey Thomas, Thanks we took a look with the team and discovered some of us were using an aws profile that lacked permissions to describe-certificate. Once that was addressed we were able to deploy with the check re-enabled. Thank you for your time.