Open agaudreault opened 6 months ago
The default value of StartTimeout
is set 1 minute when not explicitly set it ClientConfig
What does your plugin do before calling goplugin.Serve
?
@joshwizzy not much to be honest. The plugin works, when it starts correctly. You can find the code here: https://github.com/agaudreault/argo-rollouts/blob/13e8829a6691e27743f00a4ac1ec5163e51251c1/test/cmd/step-plugin-e2e/main.go.
I found out that deleting the executed binary solved the issue as well. Easier than restarting when running locally!
@agaudreault Would you mind adding an issue to the argo-rollouts repo with instructions on how to reproduce the broken state. It might be easier for folks who are interested/familiar with the argo-rollouts plugin integration to investigate. I am unfamiliar with argo-rollouts but this seems to be part of an initiative to add Step plugins to the project. Most people here may not have the required context to help out.
@joshwizzy I am the one implementing the step plugin with other maintainers hence why I'm reaching out about the behavior. This is not an argo rollout issue, but you can see the code and perhaps find out if something is wrong with how the plugin is started. It's quite simple so building a repro app should be quite easy with access to existing code.
The steps are
It seems like the go-plugin does not support to be re-executed if it is not shutdown gracefully
@agaudreault I am unable to reproduce this using the basic greeter example. I followed these steps:
comment out this line so the plugin is not killed when the client exits defer client.Kill()
build and trun the example app
./basic
2024-04-23T15:02:57.225+0300 [DEBUG] plugin: starting plugin: path=./plugin/greeter args=[./plugin/greeter]
2024-04-23T15:02:57.227+0300 [DEBUG] plugin: plugin started: path=./plugin/greeter pid=86088
2024-04-23T15:02:57.227+0300 [DEBUG] plugin: waiting for RPC address: plugin=./plugin/greeter
2024-04-23T15:02:57.237+0300 [DEBUG] plugin.greeter: message from plugin: foo=bar timestamp=2024-04-23T15:02:57.237+0300
2024-04-23T15:02:57.237+0300 [DEBUG] plugin.greeter: plugin address: address=/var/folders/sl/h6pntx3j05l0fk99605hzxy40000gn/T/plugin918908959 network=unix timestamp=2024-04-23T15:02:57.237+0300
2024-04-23T15:02:57.237+0300 [DEBUG] plugin: using plugin: version=1
2024-04-23T15:02:57.239+0300 [DEBUG] plugin.greeter: message from GreeterHello.Greet: timestamp=2024-04-23T15:02:57.239+0300
Hello!
The plugin process stays running after the client exits
ps ax |grep greeter
86088 s000 S 0:00.01 ./plugin/greeter
./basic
2024-04-23T15:04:38.176+0300 [DEBUG] plugin: starting plugin: path=./plugin/greeter args=[./plugin/greeter]
2024-04-23T15:04:38.178+0300 [DEBUG] plugin: plugin started: path=./plugin/greeter pid=86897
2024-04-23T15:04:38.178+0300 [DEBUG] plugin: waiting for RPC address: plugin=./plugin/greeter
2024-04-23T15:04:38.191+0300 [DEBUG] plugin.greeter: message from plugin: foo=bar timestamp=2024-04-23T15:04:38.191+0300
2024-04-23T15:04:38.191+0300 [DEBUG] plugin.greeter: plugin address: address=/var/folders/sl/h6pntx3j05l0fk99605hzxy40000gn/T/plugin523748303 network=unix timestamp=2024-04-23T15:04:38.191+0300
2024-04-23T15:04:38.191+0300 [DEBUG] plugin: using plugin: version=1
2024-04-23T15:04:38.193+0300 [DEBUG] plugin.greeter: message from GreeterHello.Greet: timestamp=2024-04-23T15:04:38.193+0300
Hello!
Two instances of the plugin are now running
ps ax |grep greeter
86088 s000 S 0:00.01 ./plugin/greeter
86897 s000 S 0:00.02 ./plugin/greeter
@agaudreault
Seeing as you are running on an M1 mac
Could you try running the app with this setting and see if it makes a difference
export GODEBUG=asyncpreemptoff=1
https://github.com/hashicorp/terraform-provider-aws/issues/20274 https://github.com/hashicorp/terraform-provider-aws/issues/16073 https://yaleman.org/post/2021/2021-01-01-apple-m1-terraform-and-golang/
When trying to start a plugin, it sometimes ends up in a broken state and the only way to unblock is to restart the host.
Main function trying to start the plugin
Function trying to get the plugin if it exist, and somewhat recover on error
Everytime we create a new client (restart the program), there is a new process that is created in the following state
The process is running locally on Mac
Running on version