search

pulumi / pulumi-command

Apache License 2.0
57 stars 23 forks source link

panic v3.116.1 --continue-on-error sigsegv sementation violation #435

Closed tusharshahrs closed 1 month ago

tusharshahrs commented 1 month ago

What happened?

Diagnostics:
  github:index:BranchProtection (myportal-ui):
    error: deleting urn:pulumi:github::im-github::github:index/branchProtection:BranchProtection::myportal-ui: 1 error occurred:
        * Branch protection is disabled on this repository.

  pulumi:pulumi:Stack (im-github-github):
    error: update failed
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdda2ee]
    goroutine 129 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x128dfe0?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x128f5a0, 0xc0005004e0})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc0002b3560, {0x1286d10?, 0xc0005004b0}, 0x40?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x1286d10, 0xc0005004b0}, {0xf9efe0?, 0xc000500450})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.91.1/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x1286d10, 0xc000500420}, {0xf9efe0, 0xc000500450}, 0xc000458040, 0xc000010258)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x1075560?, 0xc0002b3560}, {0x1286d10, 0xc000500420}, 0xc000254700, 0xc000341900)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.91.1/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002e8000, {0x1292b40, 0xc00050a9c0}, 0xc0002c4120, 0xc0002cc750, 0x1a79038, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1358 +0xe15
    google.golang.org/grpc.(*Server).handleStream(0xc0002e8000, {0x1292b40, 0xc00050a9c0}, 0xc0002c4120, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1735 +0x9e7
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:970 +0xbb
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 58
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:981 +0x145
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe08b6e]
    goroutine 40 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x12ca120?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x12cbb20, 0xc00045a510})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc00002f0e0, {0x12c2e70?, 0xc00045a4e0}, 0x40?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x12c2e70, 0xc00045a4e0}, {0xfd0780?, 0xc00045a480})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x12c2e70, 0xc00045a450}, {0xfd0780, 0xc00045a480}, 0xc00051c080, 0xc000298120)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x10ab160?, 0xc00002f0e0}, {0x12c2e70, 0xc00045a450}, 0xc000338280, 0xc00037e500)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc000136000, {0x12c2e70, 0xc00045a1e0}, {0x12cedc0, 0xc000582820}, 0xc0006c2360, 0xc000129740, 0x1ad8b18, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03
    google.golang.org/grpc.(*Server).handleStream(0xc000136000, {0x12cedc0, 0xc000582820}, 0xc0006c2360)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 116
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145

  github:index:BranchProtection (myportal-data):
    error: deleting urn:pulumi:github::im-github::github:index/branchProtection:BranchProtection::myportal-data: 1 error occurred:
        * Branch protection is disabled on this repository.

Resources:
    ~ .. updated
    +-.. replaced
    x2 changes. 1yy unchanged

Example

Ran the following command:

pulumi update --skip-preview -r --non-interactive --continue-on-error

Output of pulumi about

pending pulumi cli: 3.116.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

tgummerer commented 1 month ago

@tusharshahrs do you by any chance have a bit more of a reproduction than just the command? I'll try to reproduce it, but having a clear repro example would be super helpful.

kirgene commented 1 month ago

@tgummerer here are the details on the issue:

CLI          
Version      3.116.1
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  command  0.10.0
resource  github   6.2.0
language  python   unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

this happened just once after updating to the recent version, I haven't had a chance to try again.

The pulumi code creates a GitHub repo with BranchProtection resource. The BranchProtection has delete_before_replace option set

  1. run the code, it should create new resources (repo with a branch protection)
  2. disable branch protection in the GitHub repo (from the UI)
  3. modify BranchProtection resource in a way it gets recreated during the next run
  4. run the code again with this command: pulumi update --skip-preview -r --non-interactive --continue-on-error
justinvp commented 1 month ago

I's not clear to me from a cursory look if this is a --continue-on-error or pulumi-go-provider issue (or something else).

tgummerer commented 1 month ago

Thanks for the details! Having had a look at this, the stack trace looks exactly the same as https://github.com/pulumi/pulumi-go-provider/issues/186, so it looks like a pulumi-go-provider issue that has since been fixed.

Given the details above, I had a look at the providers. pulumi-github is a bridged provider and thus doesn't use pulumi-go-provider, but pulumi-command which also seems used here does, and is indeed on version 0.13.1 of pulumi-go-provider, while the fix for above issue has only landed in 0.15.0.

I'm gonna go ahead and transfer the issue over to the pulumi-command repo.

thomas11 commented 1 month ago

Hi @tusharshahrs, we just released v0.11.0 of the command provider, updated to the latest pulumi-go-provider that has the fix @tgummerer identified as a likely culprit. Could you verify whether the new version fixes the issue?

kirgene commented 1 month ago

thanks, this is what I'm getting now:

error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/home/yevhen/.pulumi/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
        return future.result()
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 85, in wait_for_rpcs
        raise exn from cause
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/rpc_manager.py", line 71, in rpc_wrapper
        result = await rpc
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 910, in do_register
        resolver = await prepare_resource(res, ty, custom, remote, props, opts, typ)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 196, in prepare_resource
        serialized_props = await rpc.serialize_properties(
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/rpc.py", line 205, in serialize_properties
        types = _types.input_type_types(typ)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/_types.py", line 508, in input_type_types
        return _types_from_py_properties(input_type_cls)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/_types.py", line 759, in _types_from_py_properties
        cls_hints = get_type_hints(prop.fget, globalns=globalns, localns=localns)
      File "/usr/lib/python3.10/typing.py", line 1871, in get_type_hints
        value = _eval_type(value, globalns, localns)
      File "/usr/lib/python3.10/typing.py", line 329, in _eval_type
        ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
      File "/usr/lib/python3.10/typing.py", line 329, in <genexpr>
        ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
      File "/usr/lib/python3.10/typing.py", line 327, in _eval_type
        return t._evaluate(globalns, localns, recursive_guard)
      File "/usr/lib/python3.10/typing.py", line 694, in _evaluate
        eval(self.__forward_code__, globalns, localns),
      File "<string>", line 1, in <module>
    AttributeError: module 'pulumi_command.common' has no attribute 'Logging'

and running another stack

Diagnostics:
  pulumi:pulumi:Stack (github-github):
    error: update failed
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe08b6e]
    goroutine 162 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x12ca120?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x12cbb20, 0xc0005485a0})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc000170630, {0x12c2e70?, 0xc000548570}, 0x40?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x12c2e70, 0xc000548570}, {0xfd0780?, 0xc000548510})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x12c2e70, 0xc0005484e0}, {0xfd0780, 0xc000548510}, 0xc00071c0c0, 0xc000010450)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x10ab160?, 0xc000170630}, {0x12c2e70, 0xc0005484e0}, 0xc0002d8600, 0xc0002a6b40)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003ce000, {0x12c2e70, 0xc000548420}, {0x12cedc0, 0xc0005744e0}, 0xc000300120, 0xc00015f260, 0x1ad8b18, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03
    google.golang.org/grpc.(*Server).handleStream(0xc0003ce000, {0x12cedc0, 0xc0005744e0}, 0xc000300120)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 161
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145

  github:index:BranchProtection (test-portal-ui):
    error: deleting urn:pulumi:github::github::github:index/branchProtection:BranchProtection::test-portal-ui: 1 error occurred:
        * Branch protection is disabled on this repository.

  github:index:BranchProtection (test-data-soonet-ui):
    error: deleting urn:pulumi:github::github::github:index/branchProtection:BranchProtection::test-data-soonet-ui: 1 error occurred:
        * Branch protection is disabled on this repository.
thomas11 commented 1 month ago

Ok, thank you for trying. The first error, 'pulumi_command.common' has no attribute 'Logging', is an independent issue #445 that affects Python programs only.

We'll continue to look into your issue.

thomas11 commented 1 month ago

The logging error is now fixed in pulumi-command v0.11.1.

lukehoban commented 1 month ago

We'll continue to look into your issue.

What issue is this tracking now? AFAICT, both of the issues noted in https://github.com/pulumi/pulumi-command/issues/435#issuecomment-2118083416 are fixed in the latest release of this provider?

thomas11 commented 1 month ago

We'll continue to look into your issue.

What issue is this tracking now? AFAICT, both of the issues noted in #435 (comment) are fixed in the latest release of this provider?

We're currently operating under the assumption that all issues are fixed but haven't had confirmation. @alayshia is checking directly with the customer. I was going to keep the issue open until we have confirmation.

thomas11 commented 1 month ago

As the issue is fixed from our point of view, I'll close this issue. Please feel free to re-open if this in error.