[Bug] 2.1.2 does not fully stop node with "chia stop all -d"

[bhorvitz]

What happened?

chia stop all -d leaves the chia_full_node_worker process running:

[chadmin@ch-n2 ~]$ chia stop all -d
chia_full_node: Stopped
Daemon stopped
[chadmin@ch-n2 ~]$ ps -efa | grep chia_full_node
chadmin     3026       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3027       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3028       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3029       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3030       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3031       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3032       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3033       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3034       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3035       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3036       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3037       1  0 06:36 ?        00:00:02 chia_full_node_worker
chadmin     3038       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3039       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3040       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3041       1  0 06:36 ?        00:00:01 chia_full_node_worker
chadmin     3050       1  0 06:36 ?        00:00:00 chia_full_node_worker
chadmin     3051       1  0 06:36 ?        00:00:00 chia_full_node_worker
[chadmin@ch-n2 ~]$ 
[chadmin@ch-n2 ~]$ chia start node
Daemon not started yet
Starting daemon
chia_full_node: started
[chadmin@ch-n2 ~]$ Traceback (most recent call last):
  File "chia/util/lock.py", line 42, in acquire
  File "filelock/_api.py", line 264, in acquire
filelock._error.Timeout: The file lock '/home/chadmin/.chia/mainnet/run/full_node.lock' could not be acquired.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "chia/server/start_service.py", line 194, in run
  File "chia/util/lock.py", line 29, in __enter__
  File "chia/util/lock.py", line 44, in acquire
chia.util.lock.LockfileError: The file lock '/home/chadmin/.chia/mainnet/run/full_node.lock' could not be acquired.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "chia/server/start_full_node.py", line 104, in <module>
  File "chia/server/start_full_node.py", line 100, in main
  File "chia/server/start_service.py", line 321, in async_run
  File "asyncio/runners.py", line 44, in run
  File "asyncio/base_events.py", line 649, in run_until_complete
  File "chia/server/start_full_node.py", line 83, in async_main
  File "chia/server/start_service.py", line 199, in run
ValueError: full_node: already running
[3918] Failed to execute script 'start_full_node' due to unhandled exception!
$

Version

2.1.2

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

$ chia start node
Daemon not started yet
Starting daemon
chia_full_node: started
$ Traceback (most recent call last):
  File "chia/server/start_full_node.py", line 104, in <module>
  File "chia/server/start_full_node.py", line 100, in main
  File "chia/server/start_service.py", line 321, in async_run
  File "asyncio/runners.py", line 44, in run
  File "asyncio/base_events.py", line 649, in run_until_complete
  File "chia/server/start_full_node.py", line 83, in async_main
  File "chia/server/start_service.py", line 195, in run
  File "contextlib.py", line 199, in __aenter__
  File "chia/server/start_service.py", line 218, in manage
  File "chia/server/server.py", line 289, in start
  File "chia/util/network.py", line 99, in create
  File "chia/util/network.py", line 111, in start
  File "aiohttp/web_runner.py", line 119, in start
  File "chia/server/chia_policy.py", line 245, in create_server
  File "chia/server/chia_policy.py", line 208, in _chia_create_server
  File "asyncio/base_events.py", line 1519, in create_server
OSError: [Errno 98] error while attempting to bind on address ('::', 8444, 0, 0): address already in use
[180062] Failed to execute script 'start_full_node' due to unhandled exception!

[bhorvitz]

Editing in console output for clarity

[github-actions[bot]]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[superadfomichev]

Bug still there , 2.1.3, alot of chia_full_node_worker's stays in RAM, preventing node(farmer) restart

[racergoodwin]

This is still relevant Ubuntu 22.04.03 Chia 2.1.3 GUI

Shutting down the GUI does not complete, stuck at "Closing down services" 32 instances of "chia_full_node_worker" still show in processes list

Killing the processes manually enables the GUI to shutdown successfully.

[quarkpapel]

Same issues here ongoing for several weeks. Node keeps crashing. Can’t reset it due to the lingering chia full node workers. kill -9 …workers. And restart farm is required every daily almost. Longest streak without crash in past month has been a few days

[wjblanke]

There seems to be a timeout in cleanup when nodes are busy with the full mempool and blocks. 2.1.4 has performance improvements which should help. Rebooting will get rid of the zombies and allow you to start and access the 8444 port.