Unable to connect to bitcoind on test_backfill_scriptpubkeys

[instagibbs]

Running on master:

lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-bcli: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-commando: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-funder: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-offers: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-pay: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-autoclean: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.594Z DEBUG   plugin-topology: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   plugin-spenderp: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   plugin-txprepare: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   plugin-bookkeeper: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   plugin-keysend: Killing plugin: exited during normal operation
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   lightningd: io_break: destroy_plugin
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   lightningd: Command returned result after jcon close
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   connectd: Shutting down
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   gossipd: Shutting down
lightningd-3 2023-01-18T14:46:54.595Z DEBUG   hsmd: Shutting down
lightningd-3 2023-01-18T14:46:54.892Z UNUSUAL plugin-bcli: bitcoin-cli -regtest -datadir=/tmp/ltests-ou2sm2rg/test_backfill_scriptpubkeys_1/lightning-3/ -rpcport=41715 -rpcuser=... -stdinrpcpass getblockhash 316 exited with status 1
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate opening fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate mutual_close fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate unilateral_close fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate delayed_to_us fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate htlc_resolution fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate penalty fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate min_acceptable fees
lightningd-3 2023-01-18T14:46:55.318Z DEBUG   lightningd: Unable to estimate max_acceptable fees
lightningd-3 2023-01-18T14:46:55.893Z UNUSUAL plugin-bcli: bitcoin-cli -regtest -datadir=/tmp/ltests-ou2sm2rg/test_backfill_scriptpubkeys_1/lightning-3/ -rpcport=41715 -rpcuser=... -stdinrpcpass getblockhash 316 exited with status 1

this continues and locally I have to pkill lightningd, though the test reports success regardless

[instagibbs]

something similar happening with test_funding_cancel_race_1 in test_connection.py as well, intermittently

[cdecker]

Hm, I had a resource leak fixture that could be used to track orphaned processes: https://github.com/ElementsProject/lightning/commit/ccd8a01f801db2d9567a65a6f2f7a7e0e9e67f84

That could catch these, as well as leaking fds, what do you think?

[instagibbs]

Added a cleaned up version to the PR, as it does report the issue when not fixed. thanks!