Golang SDK hang indefinitely

sdalezman commented 1 month ago

What happened?

I'm using the golang sdk and when running preview it just hangs forever. This only applies to resources from the Datadog package itself and not when using the aws package.

Example

Here's sample go code that when run just hangs forever:

func datadogErrorLogMonitor(ctx *pulumi.Context, serviceName string) error {
    // Create the Datadog monitor
    name := fmt.Sprintf("%sErrorLog", serviceName)
    _, err := datadog.NewMonitor(ctx, name, &datadog.MonitorArgs{
        Name:             pulumi.Sprintf("%sErrorLog", serviceName),
        Type:             pulumi.String("log alert"),
        Query:            pulumi.String("logs(\"status:error service:my-service\").index(\"*\").rollup(\"count\").last(\"5m\") > 1"),
        Message:          pulumi.String("@slack-datadog-monitor \n\nError was logged in {{log.service}} with error: {{log.attributes.error}}"),
        Tags:             pulumi.StringArray{},
        EnableLogsSample: pulumi.Bool(true),
        NotifyAudit:      pulumi.Bool(false),
        OnMissingData:    pulumi.String("default"),
        IncludeTags:      pulumi.Bool(false),
        MonitorThresholds: datadog.MonitorMonitorThresholdsArgs{
            Critical:         pulumi.String("1"),
            CriticalRecovery: pulumi.String("0"),
        },
        NotificationPresetName: pulumi.String("hide_handles"),
    })
    if err != nil {
        return err
    }
    return nil
}

Output

pulumi preview -C pulumi_v1 -s prod_v1 --diff
Previewing update (prod_v1)

View Live: https://app.pulumi.com/my-org/my-repo/my-stack/previews/my-id

 error: preview canceled

I end up cancelling the preview because minutes and minutes go by with nothing (a regular preview takes under 15 seconds to complete)

Output of `pulumi about`

CLI
Version      3.114.0
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  aws      6.32.0
resource  awsx     2.10.0
resource  datadog  4.28.0
resource  docker   4.4.3
language  go       unknown

Host
OS       darwin
Version  14.4.1
Arch     arm64

This project is written in go: executable='/opt/homebrew/bin/go' version='go version go1.22.1 darwin/arm64'

Backend
Name           pulumi.com
URL            https://app.pulumi.com/my-name
User           my-name
Organizations  my-org, my-org-1, my-org-2
Token type     personal

Dependencies:
NAME                                     VERSION
github.com/pulumi/pulumi-aws/sdk/v6      v6.32.0
github.com/pulumi/pulumi-awsx/sdk/v2     v2.10.0
github.com/pulumi/pulumi-datadog/sdk/v4  v4.28.0
github.com/pulumi/pulumi/sdk/v3          v3.113.0

Pulumi locates its logs in /var/folders/4x/l0nfshs500z95ztqxp4zjhcc0000gn/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

iwahbe commented 1 month ago

Hey @sdalezman. Thanks for filing an issue.

Can you clarify a couple things for me:

Does this happen for all datadog resources, or just datadog.Monitor?
Does this happen every time, or sometimes?
Can you include debug logs for a pulumi up that hangs? (instructions are here)

yishait commented 3 weeks ago

Hey @iwahbe

Same thing happening to me on docker container (directly on my mac its working)

tried using the example in the docs process just hangs shortly after begining of the run only when using datadog package, the strange thing is that ALL container memory during that time is being consumed

Every 2.0s: free -h                        b77aa74117d9: Thu Jun  6 17:24:14 2024

               total        used        free      shared  buff/cache   available
Mem:            15Gi       8.6Gi       3.5Gi       4.0Mi       3.5Gi       6.7Gi
Swap:             0B          0B          0B

adding pulumi about and the output for pulumi preview --logtostderr -v=9 i tried running it for a few minutes a few times it never continues beyond that point

CLI
Version      3.119.0
Go Version   go1.22.3
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  datadog  4.28.0
language  go       unknown

Host
OS       debian
Version  11.9
Arch     aarch64

This project is written in go: executable='/usr/local/bin/go' version='go version go1.22.4 linux/arm64'

Found no pending operations associated 

Backend
Name           b77aa74117d9
URL            s3://my_bucket
User           root
Organizations
Token type     personal

Dependencies:
NAME                                     VERSION
github.com/pulumi/pulumi-datadog/sdk/v4  v4.28.0
github.com/pulumi/pulumi/sdk/v3          v3.119.0

Pulumi locates its logs in /tmp by default

root@b77aa74117d9:/app/app# pulumi preview --logtostderr -v=9 > docker_log.txt
I0606 17:23:15.310286    7258 update.go:206] *** Starting Update(preview=true) ***
I0606 17:23:15.311258    7258 plugins.go:124] gatherPluginsFromProgram(): gathering plugins from language host
I0606 17:23:15.311941    7258 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0606 17:23:15.312540    7258 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0606 17:23:15.312562    7258 plugin.go:186] newPlugin(): Launching plugin 'go' from '/pulumi/bin/pulumi-language-go' with args: -root=/app/app,127.0.0.1:43319
I0606 17:23:15.341700    7258 langruntime_plugin.go:249] langhost[go].GetPluginInfo() executing
I0606 17:23:15.342200    7258 langruntime_plugin.go:136] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) executing
I0606 17:23:15.398710    7258 langruntime_plugin.go:186] langhost[go].GetRequiredPlugins(root=/app/app, program=/app/app, entryPoint=.) success: #versions=1
I0606 17:23:15.398748    7258 plugins.go:137] gatherPluginsFromProgram(): plugin go <nil> () is required by language host
I0606 17:23:15.398769    7258 plugins.go:137] gatherPluginsFromProgram(): plugin datadog 4.28.0 () is required by language host
I0606 17:23:15.398782    7258 plugins.go:149] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I0606 17:23:15.398810    7258 plugins.go:193] ensurePluginsAreInstalled(): beginning
I0606 17:23:15.398828    7258 plugins.go:1800] GetPluginPath(language, go, <nil>): found on $PATH /pulumi/bin/pulumi-language-go
I0606 17:23:15.398849    7258 plugins.go:1823] GetPluginPath(language, go, <nil>): found next to current executable /pulumi/bin/pulumi-language-go
I0606 17:23:15.398856    7258 plugins.go:203] ensurePluginsAreInstalled(): plugin go <nil> already installed
I0606 17:23:15.399071    7258 plugins.go:1874] GetPluginPath(resource, datadog, 4.28.0): enabling new plugin behavior
I0606 17:23:15.399117    7258 plugins.go:2008] SelectCompatiblePlugin(..., datadog): beginning
I0606 17:23:15.399123    7258 plugins.go:2041] SelectCompatiblePlugin(..., datadog): skipping plugin datadog-3.3.0: semver mismatch
I0606 17:23:15.399126    7258 plugins.go:2037] SelectCompatiblePlugin(..., datadog): best plugin datadog-4.28.0: semver match
I0606 17:23:15.399278    7258 plugins.go:1883] GetPluginPath(resource, datadog, 4.28.0): found in cache at /root/.pulumi/plugins/resource-datadog-v4.28.0/pulumi-resource-datadog
I0606 17:23:15.399479    7258 plugins.go:203] ensurePluginsAreInstalled(): plugin datadog 4.28.0 already installed
I0606 17:23:15.399486    7258 plugins.go:236] ensurePluginsAreInstalled(): completed
I0606 17:23:15.399495    7258 plugins.go:340] computeDefaultProviderPlugins(): considering go
I0606 17:23:15.399503    7258 plugins.go:343] computeDefaultProviderPlugins(): skipping go, not a resource provider
I0606 17:23:15.399507    7258 plugins.go:340] computeDefaultProviderPlugins(): considering datadog-4.28.0
I0606 17:23:15.399513    7258 plugins.go:371] computeDefaultProviderPlugins(): plugin datadog-4.28.0 selected for package datadog (first seen)
I0606 17:23:15.399520    7258 log.go:73] computeDefaultProviderPlugins(): summary of default plugins:
I0606 17:23:15.399773    7258 plugins.go:379]   datadog         = 4.28.0
I0606 17:23:15.401110    7258 deployment_executor.go:218] deploymentExecutor.Execute(...): waiting for incoming events
I0606 17:23:15.401385    7258 rpc.go:78] Marshaling property for RPC[]: pulumi:tags={map[pulumi:template:{go}]}
I0606 17:23:15.401392    7258 rpc.go:78] Marshaling property for RPC[]: pulumi:template={go}
I0606 17:23:15.401684    7258 step_executor.go:513] StepExecutor worker(0): worker coming online
I0606 17:23:15.401722    7258 step_executor.go:513] StepExecutor worker(0): worker waiting for incoming chains
I0606 17:23:15.401728    7258 step_executor.go:513] StepExecutor worker(1): worker coming online
I0606 17:23:15.401730    7258 step_executor.go:513] StepExecutor worker(1): worker waiting for incoming chains
I0606 17:23:15.401734    7258 step_executor.go:513] StepExecutor worker(2): worker coming online
I0606 17:23:15.401736    7258 step_executor.go:513] StepExecutor worker(2): worker waiting for incoming chains
I0606 17:23:15.401739    7258 step_executor.go:513] StepExecutor worker(3): worker coming online
I0606 17:23:15.401742    7258 step_executor.go:513] StepExecutor worker(3): worker waiting for incoming chains
I0606 17:23:15.401746    7258 step_executor.go:513] StepExecutor worker(4): worker coming online
I0606 17:23:15.401748    7258 step_executor.go:513] StepExecutor worker(4): worker waiting for incoming chains
I0606 17:23:15.401842    7258 step_executor.go:513] StepExecutor worker(5): worker coming online
I0606 17:23:15.401849    7258 step_executor.go:513] StepExecutor worker(5): worker waiting for incoming chains
I0606 17:23:15.401853    7258 step_executor.go:513] StepExecutor worker(6): worker coming online
I0606 17:23:15.401858    7258 step_executor.go:513] StepExecutor worker(6): worker waiting for incoming chains
I0606 17:23:15.401863    7258 step_executor.go:513] StepExecutor worker(7): worker coming online
I0606 17:23:15.401865    7258 step_executor.go:513] StepExecutor worker(7): worker waiting for incoming chains
I0606 17:23:15.401870    7258 step_executor.go:513] StepExecutor worker(8): worker coming online
I0606 17:23:15.401872    7258 step_executor.go:513] StepExecutor worker(8): worker waiting for incoming chains
I0606 17:23:15.401876    7258 step_executor.go:513] StepExecutor worker(9): worker coming online
I0606 17:23:15.401879    7258 step_executor.go:513] StepExecutor worker(9): worker waiting for incoming chains
I0606 17:23:15.401882    7258 step_executor.go:513] StepExecutor worker(10): worker coming online
I0606 17:23:15.401884    7258 step_executor.go:513] StepExecutor worker(10): worker waiting for incoming chains
I0606 17:23:15.401889    7258 step_executor.go:513] StepExecutor worker(11): worker coming online
I0606 17:23:15.401892    7258 step_executor.go:513] StepExecutor worker(11): worker waiting for incoming chains
I0606 17:23:15.401895    7258 step_executor.go:513] StepExecutor worker(12): worker coming online
I0606 17:23:15.401897    7258 step_executor.go:513] StepExecutor worker(12): worker waiting for incoming chains
I0606 17:23:15.401901    7258 step_executor.go:513] StepExecutor worker(13): worker coming online
I0606 17:23:15.401916    7258 step_executor.go:513] StepExecutor worker(13): worker waiting for incoming chains
I0606 17:23:15.401923    7258 step_executor.go:513] StepExecutor worker(14): worker coming online
I0606 17:23:15.402118    7258 step_executor.go:513] StepExecutor worker(14): worker waiting for incoming chains
I0606 17:23:15.402135    7258 step_executor.go:513] StepExecutor worker(15): worker coming online
I0606 17:23:15.402145    7258 step_executor.go:513] StepExecutor worker(15): worker waiting for incoming chains

guineveresaenger commented 3 weeks ago

Hi folks!

@sdalezman - would you be able to help us by addressing @iwahbe's questions?

pulumi / pulumi-datadog