AWS API Gateway create issue - “apigateway.eu-west-1.amazonaws.com” certificate is not standards compliant

[janaka]

What happened?

I'm trying to standup an AWS API Gateway with a Lambda attached. Paired it back to the basic example from docs as below. The create step hangs with errors visible in debug mode. Have tried upgrading to the latest bits of Pulumi with not luck. I now suspect that it's a macOS + Golang level issue with how certs are handled based on this GH Issues.

I'm brand new to Pulumi so don't have other working examples to compare behaviour.

Steps to reproduce

import * as aws from "@pulumi/aws"; // v5.23.0
import * as awsx from "@pulumi/awsx"; // v1.0.1
import * as apigateway from "@pulumi/aws-apigateway"; // v1.0.1

let endpoint = new apigateway.RestAPI("example", {
// let endpoint = new awsx.classic.apigateway.API("example", { // same error 
  routes: [{
      path: "/",
      method: "GET",
      eventHandler: new aws.lambda.CallbackFunction("test", {
          memorySize: 256,
          callback: async (event) => {
              return {
                  statusCode: 200,
                  body: "<h1>Hello world!</h1>",
              };
          },
      }),
  }],
})

Expected Behavior

A succesful creation of an API Gateway instance visible in the AWS Console.

Actual Behavior

Hangs on the gateway creation step. With verbose level 11 and debug switched on the following error message is revealed.

    debug: -----------------------------------------------------
    debug: [aws-sdk-go] DEBUG: Send Request apigateway/CreateRestApi failed, attempt 8/25, error RequestError: send request failed
    debug: caused by: Post "https://apigateway.eu-west-1.amazonaws.com/restapis": x509: “apigateway.eu-west-1.amazonaws.com” certificate is not standards compliant
    debug: [aws-sdk-go] DEBUG: Retrying Request apigateway/CreateRestApi, attempt 9
    debug: [aws-sdk-go] DEBUG: Request apigateway/CreateRestApi Details:
    debug: ---[ REQUEST POST-SIGN ]-----------------------------
    debug: POST /restapis HTTP/1.1
    debug: Host: apigateway.eu-west-1.amazonaws.com
    debug: User-Agent: APN/1.0 Pulumi/1.0 Pulumi/1.0 Pulumi-Aws/0.11+compatible (+https://www.pulumi.com) aws-sdk-go/1.44.150 (go1.19.3; darwin; amd64)
    debug: Content-Length: 55
    debug: Accept: application/json
    debug: Authorization: AWS4-HMAC-SHA256 Credential=AKIA36YS4V7HAIEH/20221212/eu-west-1/apigateway/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=8e995fb3b8d664839dc74a00708b5b119c252f684114945967ce29f327
    debug: Content-Type: application/json
    debug: X-Amz-Date: 20221212T171914Z
    debug: Accept-Encoding: gzip
    debug: 
    debug: {"binaryMediaTypes":["*/*"],"name":"example","tags":{}}
    debug: -----------------------------------------------------

Output of pulumi about

CLI          
Version      3.49.0
Go Version   go1.19.3
Go Compiler  gc

Plugins
NAME            VERSION
aws             5.23.0
aws-apigateway  1.0.1
aws-native      0.44.0
awsx            1.0.1
docker          3.6.1
nodejs          unknown

Host     
OS       darwin
Version  12.3.1
Arch     x86_64

This project is written in nodejs: executable='/usr/local/bin/node' version='v19.1.0'

Current Stack: janaka-org/redwood-todo-app/dev

TYPE                                               URN
pulumi:pulumi:Stack                                urn:pulumi:dev::redwood-todo-app::pulumi:pulumi:Stack::redwood-todo-app-dev
pulumi:providers:aws                               urn:pulumi:dev::redwood-todo-app::pulumi:providers:aws::default_5_23_0
aws:iam/role:Role                                  urn:pulumi:dev::redwood-todo-app::aws:iam/role:Role::test
aws:lambda/function:Function                       urn:pulumi:dev::redwood-todo-app::aws:lambda/function:Function::test
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-1b4caae3
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-e1a3786d
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-019020e7
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-74d12784
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-6c156834
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-a1de8170
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-7cd09230
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-4aaabb8e
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::redwood-todo-app::aws:iam/rolePolicyAttachment:RolePolicyAttachment::test-b5aeb6b6
pulumi:providers:aws-apigateway                    urn:pulumi:dev::redwood-todo-app::pulumi:providers:aws-apigateway::default_1_0_1
pulumi:providers:pulumi                            urn:pulumi:dev::redwood-todo-app::pulumi:providers:pulumi::default
aws-apigateway:index:RestAPI                       urn:pulumi:dev::redwood-todo-app::aws-apigateway:index:RestAPI::example
pulumi:providers:aws                               urn:pulumi:dev::redwood-todo-app::pulumi:providers:aws::default_5_16_2
aws:apigateway:x:API                               urn:pulumi:dev::redwood-todo-app::aws:apigateway:x:API::example

Found no pending operations associated with janaka-org/dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/janaka
User           janaka
Organizations  janaka, janaka-org

Dependencies:
NAME                    VERSION
@pulumi/aws             5.23.0
@pulumi/aws-apigateway  1.0.1
@pulumi/aws-native      0.44.0
@pulumi/awsx            1.0.1
@pulumi/pulumi          3.49.0
@types/node             14.18.34

Pulumi locates its logs in /var/folders/fq/8mlxgplx0tq08m3_4b06ff4h0000gn/T/ by default

Additional context

This issue might be related https://github.com/golang/go/issues/51991. It's the closest I could find Googling.

Update: I don't get this problem when doing pulumi up using the pulumi/pulumi-nodejs container.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[AaronFriel]

Could you check if updating to the latest version of macOS resolves this issue for you?

[squaremo]

As a side note -- thanks @janaka for digging into the issue, and finding a repro -- makes our job easier!

(I'm removing needs-triage since I think we're past that now)

[janaka]

Could you check if updating to the latest version of macOS resolves this issue for you?

@AaronFriel will try next week when I'm back on a stronger Internet connection. I did try, was having issue with downloads.

[AaronFriel]

Thanks @janaka, as you might have seen we've encountered this issue internally, but haven't been able to root cause. Our working theory is that this is an interaction between the Go internals and out of date macOS versions.

[mikhailshilkov]

Since this was quite a while ago and no new comments were posted, I'll go ahead and close this issue as stale. Please open a new issue if the problem still persists. Thank you!