pulumi / pulumi-digitalocean

A DigitalOcean Pulumi resource package, providing multi-language access to DigitalOcean
Apache License 2.0
81 stars 13 forks source link

pulumi up results in: `error reading from server: read tcp 127.0.0.1:63025->127.0.0.1:63022: wsarecv: An existing connection was forcibly closed by the remote host.` #658

Open douglasg14b opened 4 months ago

douglasg14b commented 4 months ago

What happened?

When I try an run puluumi up from a rpeviously working configuration. I get the following exception:

Diagnostics: digitalocean:index:App (hiveAppService): error: error reading from server: read tcp 127.0.0.1:63025->127.0.0.1:63022: wsarecv: An existing connection was forcibly closed by the remote host.

pulumi:pulumi:Stack (hive-npr): panic: runtime error: index out of range [0] with length 0 goroutine 48 [running]: github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppCORSPolicy({0xc000c7d100?, 0xc000f03170?, 0x1f8a416?}) /home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:2034 +0xb45 github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppIngress({0xc000c7d0f0, 0xc000d44b70?, 0x1f8eea9?}) /home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:2122 +0x72c github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppSpec({0xc0010872a0?, 0x1f899ea?, 0xc0002b7650?}) /home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:1027 +0xc05 github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.resourceDigitalOceanAppUpdate({0x24999e8, 0xc0002b7650}, 0xc000d3b180, {0x1d91d60?, 0xc0006263f0?}) /home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/resource_app.go:149 +0xdd github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(Resource).update(0xc0003121c0, {0x2499940, 0xc000889470}, 0xd?, {0x1d91d60, 0xc0006263f0}) /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240202163305-e2a20ae13ef9/helper/schema/resource.go:817 +0x11b github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(Resource).Apply(0xc0003121c0, {0x2499940, 0xc000889470}, 0xc0007c1930, 0xc0005a6c80, {0x1d91d60, 0xc0006263f0}) /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240202163305-e2a20ae13ef9/helper/schema/resource.go:924 +0x83a github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000620420?, {0x0?, 0x1d?, 0xffffffffffffffff?}}, {0x2499940, 0xc000889470}, {0x1f9c8d1, 0x10}, {0x2499fa8, 0xc00072ede0}, ...) /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.75.0/pkg/tfshim/sdk-v2/provider.go:122 +0x165 github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(Provider).Update(0xc000533980, {0x2499940?, 0xc000888c30?}, 0xc0004f2090) /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.75.0/pkg/tfbridge/provider.go:1268 +0xbee github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0x2499940, 0xc000888c30}, {0x1eecb20?, 0xc0004f2090}) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.105.0/proto/go/provider_grpc.pb.go:629 +0x75 github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2499940, 0xc00051ca50}, {0x1eecb20, 0xc0004f2090}, 0xc000710aa0, 0xc000f882a0) /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0 github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0x1f526c0?, 0xc000533980}, {0x2499940, 0xc00051ca50}, 0xc000d3a080, 0xc000a7b5c0) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.105.0/proto/go/provider_grpc.pb.go:631 +0x135 google.golang.org/grpc.(Server).processUnaryRPC(0xc000608a00, {0x2499940, 0xc00051c8d0}, {0x24a34c0, 0xc00070a4e0}, 0xc0008a4120, 0xc0001cdaa0, 0x30ef930, 0x0) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1383 +0xe03 google.golang.org/grpc.(Server).handleStream(0xc000608a00, {0x24a34c0, 0xc00070a4e0}, 0xc0008a4120) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1794 +0x100c google.golang.org/grpc.(Server).serveStreams.func2.1() /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1027 +0x8b created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 55 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1038 +0x135

error: update failed

Example

Unknown, this started happening out of the blue. I don't know how to repro it, or where I would even start.

Output of pulumi about

CLI
Version      3.109.0
Go Version   go1.22.0
Go Compiler  gc

Plugins
NAME          VERSION
digitalocean  4.25.2
nodejs        unknown

Host
OS       Microsoft Windows 10 Pro
Version  10.0.19045 Build 19045
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v20.10.0'

Current Stack: douglasg14b/hive/npr

TYPE                                                                            URN
pulumi:pulumi:Stack                                                             urn:pulumi:npr::hive::pulumi:pulumi:Stack::hive-npr
pulumi:providers:digitalocean                                                   urn:pulumi:npr::hive::pulumi:providers:digitalocean::default_4_25_2
digitalocean:index/spacesBucket:SpacesBucket                                    urn:pulumi:npr::hive::digitalocean:index/spacesBucket:SpacesBucket::hiveThumbnailsBucket
digitalocean:index/vpc:Vpc                                                      urn:pulumi:npr::hive::digitalocean:index/vpc:Vpc::hiveServiceVpc
digitalocean:index/spacesBucket:SpacesBucket                                    urn:pulumi:npr::hive::digitalocean:index/spacesBucket:SpacesBucket::hiveCreativesBucket
digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration  urn:pulumi:npr::hive::digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration::hiveCreativesBucketCors
digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration  urn:pulumi:npr::hive::digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration::hiveThumbnailsBucketCors
digitalocean:index/databaseCluster:DatabaseCluster                              urn:pulumi:npr::hive::digitalocean:index/databaseCluster:DatabaseCluster::hiveDatabaseCluster
digitalocean:index/databaseUser:DatabaseUser                                    urn:pulumi:npr::hive::digitalocean:index/databaseUser:DatabaseUser::hideDatabaseUser
digitalocean:index/databaseDb:DatabaseDb                                        urn:pulumi:npr::hive::digitalocean:index/databaseDb:DatabaseDb::hiveDatabase
digitalocean:index/app:App                                                      urn:pulumi:npr::hive::digitalocean:index/app:App::hiveAppService

Found no pending operations associated with npr

Backend
Name           pulumi.com
URL            https://app.pulumi.com/douglasg14b
User           douglasg14b
Organizations  douglasg14b
Token type     personal

Pulumi locates its logs in C:\Users\Douglas\AppData\Local\Temp by default

Additional context

My deployments where working, but not they do not. Unsure if this is a local machine problem, or a problem with pulimi? Either way, there's probably a location here for error handling.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

iwahbe commented 4 months ago

Hey @douglasg14b. I'm sorry you're hitting that, it looks like something has gone wrong here. If you could provide us a program that reproduces the panic, we would be happy to take a look.

Unfortunately, it's very hard for us to fix the problem if we can't reproduce it.

douglasg14b commented 4 months ago

I'm having trouble reproing this anywhere else. Is there more information I might be able to provide that may lead to this being more approachable by you or your team?

iwahbe commented 3 months ago

Sorry for the long wait time. If you run PULUMI_DEBUG_GRPC=logs.json pulumi up, then send us the resulting logs.json file that could help.

That json file might contain secrets, depending on how you have configured your provider. Make sure to scrub those.

mdegel commented 3 months ago

Might be unrelated, but I had a similar issue with pulumi-artifactory (Python):

With pulumi_artifactory==6.5.1 I get similar errors:

$ pulumi up
# ...
  artifactory:index:LocalGenericRepository (demo-staging-generic-local):
    error: error reading from server: read tcp 127.0.0.1:60632->127.0.0.1:60629: wsarecv: An existing connection was forcibly closed by the remote host.

Reverting to an older version pulumi_artifactory==6.4.5 yields actually more information about then real issue:

$ pulumi up
# ...
  artifactory:index:LocalGenericRepository (demo-staging-generic-local):
    error: 1 error occurred:
        *
    400 PUT https://my.url.com/artifactory/api/repositories/demo-staging-generic-local
    {
      "errors" : [ {
        "status" : 400,
        "message" : "This REST API is available only in Artifactory Pro (see: jfrog.com/artifactory/features). If you are already running Artifactory Pro please make sure your server is activated with a valid license key.\n"
      } ]
    }

In my case the newer versions of the generated pulumi libraries started hiding the actual internal errors (which themselves are unrelated to pulumi). Might be a similar issue here with unexpected responses from digitalocean?

douglasg14b commented 1 month ago

@iwahbe

This happens when i have an app spec with CORS rules (Which even has a placeholder example header 😅):

When the CORS rules are removed, this issue no longer occurs.

new digitalocean.App('[REDACTED]', {
        spec: {
            name: '[REDACTED]',
            region: digitalocean.Region.NYC3,
            databases: [{
                name: '[REDACTED]',
                clusterName: '[REDACTED]',
                dbName: '[REDACTED]',
                dbUser: '[REDACTED]',
                production: true,
            }],
            services: [{
                name: '[REDACTED]',
                instanceSizeSlug: 'professional-xs',
                github: {
                    branch: '[REDACTED]',
                    repo: '[REDACTED]',
                    deployOnPush: false,
                },
                sourceDir: '[REDACTED]',
                dockerfilePath: '[REDACTED]',
                httpPort: 8080,
            }],
            staticSites: [{
                name: 'client-app',
                dockerfilePath: 'Dockerfile.client',
                outputDir: '/app',
                github: {
                    branch: '[REDACTED]',
                    repo: '[REDACTED]',
                    deployOnPush: false,
                },
                catchallDocument: 'index.html',
            }],
            ingress: {
                rules: [
                    {
                        component: {
                            name: '[REDACTED]',
                            preservePathPrefix: true,
                        },
                        match: {
                            path: {
                                prefix: '/api',
                            },
                        },
                        cors: {
                            allowMethods: ['GET', 'POST', 'PUT', 'DELETE'],
                            allowHeaders: ['*'],
                            exposeHeaders: ['X-My-Custom-Header'],
                            allowCredentials: false,
                        },
                    },
                    {
                        component: {
                            name: 'client-app',
                        },
                        match: {
                            path: {
                                prefix: '/',
                            },
                        },
                    },
                ],
            },
        },
    });
iwahbe commented 1 month ago

Hey @douglasg14b. Thanks for the repro. Amazingly, the code you gave actually works as given (except the reference to the free variable envs, which can be removed).

For a simplified repro:

import * as digitalocean from "@pulumi/digitalocean";

new digitalocean.App('[REDACTED]', {
  spec: {
    name: '[REDACTED]',
    region: digitalocean.Region.NYC3,
    ingress: {
      rules: [
        {
          cors: {
            allowMethods: ['GET', 'POST', 'PUT', 'DELETE'],
          },
        },
      ],
    },
  },
});
iwahbe commented 1 month ago

I have confirmed that this is a bug upstream and have opened an issue there: https://github.com/digitalocean/terraform-provider-digitalocean/issues/1176. We will take the fix as soon as they do.

douglasg14b commented 1 month ago

Interesting, thank you for finding & confirming the issue.