op-node does not respond to `SIGINT` or `SIGTERM`

ethereum-optimism / optimism

Optimism is Ethereum, scaled.

https://optimism.io

MIT License

5.64k stars 3.27k forks source link

op-node does not respond to `SIGINT` or `SIGTERM` #8086

Closed yorickdowne closed 11 months ago

yorickdowne commented 1 year ago

Describe the bug

The op-node process does not shut down when sent a SIGINT or SIGTERM. It does shut down immediately when set a SIGHUP.

To Reproduce

Run op-node, in or out of Docker. Verify it runs directly, without a shell wrapper (either direct to begin with or via exec)
Observe logs
kill -s SIGTERM pid-of-op-node No shutdown
kill -s SIGINT pid-of-op-node No shutdown
kill -s SIGHUP pid-of-op-node Immediate shutdown

Expected behavior

op-node shuts down on SIGINT and SIGTERM

System Specs:

OS: Linux
Package Version (or commit hash): v0.0.0-74ff044a-1699318761

ajsutton commented 1 year ago

Thanks for the report @yorickdowne - testing locally with the latest develop I can see some interesting results. It seems to respond to SIGTERM and shutdown the p2p processes but other parts don't seem to notice so we may be missing a context check somewhere like in the derivation driver.

I also got one case where it panicked because of a closed channel which I suspect is a rare race condition:

panic: send on closed channel

goroutine 360 [running]:
github.com/ethereum-optimism/optimism/op-node/p2p.(*NodeP2P).DiscoveryProcess.func5()
    /Users/aj/Documents/code/optimism/op-node/p2p/discovery.go:333 +0x100
created by github.com/ethereum-optimism/optimism/op-node/p2p.(*NodeP2P).DiscoveryProcess in goroutine 388
    /Users/aj/Documents/code/optimism/op-node/p2p/discovery.go:326 +0x54c

Will have to do some more digging.