pulumi / pulumi-kubernetes-operator

A Kubernetes Operator that automates the deployment of Pulumi Stacks
Apache License 2.0
226 stars 54 forks source link

Report status when stack is locked #736

Open EronWright opened 3 weeks ago

EronWright commented 3 weeks ago

A given stack may become stuck due to locking. Can the operator reflect this condition to the user? Perhaps the update could have a Reason for its Failed condition, that would help the Stack controller produce a better status.

Here's an example of how the problem may occur.

 pulumi 2024-10-03T00:08:30.369Z INFO    pulumi  Updating (dev)
 pulumi 2024-10-03T00:08:30.370Z INFO    pulumi
 pulumi 2024-10-03T00:08:45.917Z WARN    pulumi  error: [500] Internal Server Error
 pulumi 2024-10-03T00:08:45.995Z ERROR   server  up completed with an error      {"error": "failed to run update: exit status 255\ncode: 255\nstdout: Updating (dev)\n\n\nstderr: error: [500] Internal Server Error\n\n"}
 pulumi 2024-10-03T00:08:45.996Z ERROR   cmd.serve.grpc  finished streaming call with code Unknown       {"grpc.start_time": "2024-10-03T00:08:28Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "Up", "peer.address": "10.1.33.119:44098", "error": "failed to run update: exit status 255\ncode: 255\nstdout: Updating (dev)\n\n\nstderr: error: [500] Internal Server Error\n\n", "grpc.code": "Unknown", "grpc.time_ms": 17067.732}
random-yaml-workspace-0 pulumi 2024-10-03T00:08:46.868Z INFO    server  selected a stack        {"name": "dev"}
random-yaml-workspace-0 pulumi 2024-10-03T00:08:48.256Z INFO    cmd.serve.grpc  finished unary call with code OK        {"grpc.start_time": "2024-10-03T00:08:46Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "SelectStack", "peer.address": "10.1.33.119:44478", "grpc.code": "OK", "grpc.time_ms": 2078.996}
 pulumi 2024-10-03T00:08:49.541Z INFO    pulumi  Updating (dev)
 pulumi 2024-10-03T00:08:49.542Z INFO    pulumi
 pulumi 2024-10-03T00:08:49.895Z WARN    pulumi  error: [409] another update is currently in progress or was interrupted. Try again later.) Try again later
 pulumi 2024-10-03T00:08:49.895Z WARN    pulumi      for more information about this error, see https://pulumi.io/oa5g1o
 pulumi 2024-10-03T00:08:49.895Z WARN    pulumi  To learn more about possible reasons and resolution, visit https://www.pulumi.com/docs/troubleshooting/#conflict

Leading to:

 pulumi 2024-10-03T00:08:53.653Z INFO    pulumi  Updating (dev)
 pulumi 2024-10-03T00:08:53.654Z INFO    pulumi
 pulumi 2024-10-03T00:08:53.960Z WARN    pulumi  error: [409] another update is currently in progress or was interrupted. Try again later.) Try again later
 pulumi 2024-10-03T00:08:53.961Z WARN    pulumi      for more information about this error, see https://pulumi.io/oa5g1o
 pulumi 2024-10-03T00:08:53.962Z WARN    pulumi  To learn more about possible reasons and resolution, visit https://www.pulumi.com/docs/troubleshooting/#conflict
cleverguy25 commented 3 weeks ago

Added to epic https://github.com/pulumi/pulumi-kubernetes-operator/issues/586