search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.83k stars 2.03k forks source link

[Bug] full_node asyncio : ERROR Task was destroyed but it is pending! #18343

Closed hajes closed 1 month ago

hajes commented 1 month ago

What happened?

Check out log output guys because there are so many errors, no idea what is exactly issue.

I have installed a new rig with PCIe 4 to handle modern GPUs.

Node runs latest Debian Bookworm.

I have tried Chia Debian version...crashes randomly after hours or day of farming. Removed the Debian version of Chia, and installed from Git source because I have used it in old server for years without issues.

Everything has been made fresh new, including initial sync. This time, farming run about 10 hours, and crashed again with what seems to be same errors.

Systemd suppose to restart failed process...what is interesting systemd claims status RUNNING, all green no errors...but everything related to chia is dead...chia show -s or chia farm summary no response.

Any suggestions why, please? The only thing that changed is latest version of Chia 2.4.1 > 2.4.2

After manual systemctl restart chia-farmer.service, everything runs again until it crashes later on.

Version

2.4.2

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2024-07-19T23:10:02.400 full_node chia.full_node.full_node: ERROR    Invalid block from peer: PeerInfo(_ip=IPv4Address('107.192.49.134'), _port=28744) Err.INVALID_RC_IP_VDF
2024-07-19T23:10:02.402 full_node chia.full_node.full_node: ERROR    sync from fork point failed: ValueError: Failed to validate block batch 5666278 to 5666309
2024-07-19T23:10:02.410 full_node chia.full_node.full_node: ERROR    failed fetching 5666406 to 5666437 from peers
2024-07-19T23:11:28.946 full_node chia.types.blockchain_format.vdf: ERROR    ['  File "/home/hajes/chia-blockchain/venv/bin/chia_full_node", line 8, in <module>\n    sys.exit(main())\n', '  File "/home/hajes/chia-blockchain/chia/server/start_full_node.py", line 102, in main\n    return async_run(coro=async_main(service_config), connection_limit=target_peer_count)\n', '  File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 323, in async_run\n    return asyncio.run(coro)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run\n    return runner.run(main)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run\n    return self._loop.run_until_complete(task)\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete\n    self.run_forever()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 607, in run_forever\n    self._run_once()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once\n    handle._run()\n', '  File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run\n    self._context.run(self._callback, *self._args)\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1138, in validate_block_batches\n    success, state_change_summary, err = await self.add_block_batch(\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1281, in add_block_batch\n    pre_validation_results: List[PreValidationResult] = await self.blockchain.pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/blockchain.py", line 805, in pre_validate_blocks_multiprocessing\n    return await pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 345, in pre_validate_blocks_multiprocessing\n    asyncio.get_running_loop().run_in_executor(\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor\n    executor.submit(func, *args), loop=self)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 783, in submit\n    self._start_executor_manager_thread()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 722, in _start_executor_manager_thread\n    self._launch_processes()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 749, in _launch_processes\n    self._spawn_process()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 759, in _spawn_process\n    p.start()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.11/multiprocessing/context.py", line 281, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 107, in batch_pre_validate_blocks\n    required_iters, error = validate_finished_header_block(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/block_header_validation.py", line 975, in validate_finished_header_block\n    if not validate_vdf(\n', '  File "/home/hajes/chia-blockchain/chia/types/blockchain_format/vdf.py", line 61, in validate_vdf\n    tb = traceback.format_stack()\n'] INVALID VDF INFO. Have: VDFInfo { challenge: 63965ea158e233000b1a169cbc0c589ae37c089856d396ab14fff9822abbca59, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } } Expected: VDFInfo { challenge: 4ea0308de61a8182cd172ff314f61e3545fb3fd60fd431a6583c76e94e461055, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } }
2024-07-19T23:11:30.029 full_node chia.full_node.full_node: ERROR    Invalid block from peer: PeerInfo(_ip=IPv4Address('38.207.131.136'), _port=8444) Err.INVALID_RC_IP_VDF
2024-07-19T23:11:30.031 full_node chia.full_node.full_node: ERROR    sync from fork point failed: ValueError: Failed to validate block batch 5666278 to 5666309
2024-07-19T23:11:30.038 full_node chia.full_node.full_node: ERROR    failed fetching 5666374 to 5666405 from peers
2024-07-19T23:22:22.243 full_node chia.types.blockchain_format.vdf: ERROR    ['  File "/home/hajes/chia-blockchain/venv/bin/chia_full_node", line 8, in <module>\n    sys.exit(main())\n', '  File "/home/hajes/chia-blockchain/chia/server/start_full_node.py", line 102, in main\n    return async_run(coro=async_main(service_config), connection_limit=target_peer_count)\n', '  File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 323, in async_run\n    return asyncio.run(coro)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run\n    return runner.run(main)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run\n    return self._loop.run_until_complete(task)\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete\n    self.run_forever()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 607, in run_forever\n    self._run_once()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once\n    handle._run()\n', '  File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run\n    self._context.run(self._callback, *self._args)\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1138, in validate_block_batches\n    success, state_change_summary, err = await self.add_block_batch(\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1281, in add_block_batch\n    pre_validation_results: List[PreValidationResult] = await self.blockchain.pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/blockchain.py", line 805, in pre_validate_blocks_multiprocessing\n    return await pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 345, in pre_validate_blocks_multiprocessing\n    asyncio.get_running_loop().run_in_executor(\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor\n    executor.submit(func, *args), loop=self)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 783, in submit\n    self._start_executor_manager_thread()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 722, in _start_executor_manager_thread\n    self._launch_processes()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 749, in _launch_processes\n    self._spawn_process()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 759, in _spawn_process\n    p.start()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.11/multiprocessing/context.py", line 281, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 107, in batch_pre_validate_blocks\n    required_iters, error = validate_finished_header_block(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/block_header_validation.py", line 975, in validate_finished_header_block\n    if not validate_vdf(\n', '  File "/home/hajes/chia-blockchain/chia/types/blockchain_format/vdf.py", line 61, in validate_vdf\n    tb = traceback.format_stack()\n'] INVALID VDF INFO. Have: VDFInfo { challenge: 63965ea158e233000b1a169cbc0c589ae37c089856d396ab14fff9822abbca59, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } } Expected: VDFInfo { challenge: 4ea0308de61a8182cd172ff314f61e3545fb3fd60fd431a6583c76e94e461055, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } }
2024-07-19T23:22:23.297 full_node chia.full_node.full_node: ERROR    Invalid block from peer: PeerInfo(_ip=IPv4Address('73.177.1.221'), _port=8444) Err.INVALID_RC_IP_VDF
2024-07-19T23:22:23.300 full_node chia.full_node.full_node: ERROR    sync from fork point failed: ValueError: Failed to validate block batch 5666278 to 5666309
2024-07-19T23:22:23.830 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
2024-07-19T23:23:02.724 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
2024-07-19T23:39:22.028 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
2024-07-19T23:49:26.214 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
2024-07-19T23:59:30.082 full_node chia.types.blockchain_format.vdf: ERROR    ['  File "/home/hajes/chia-blockchain/venv/bin/chia_full_node", line 8, in <module>\n    sys.exit(main())\n', '  File "/home/hajes/chia-blockchain/chia/server/start_full_node.py", line 102, in main\n    return async_run(coro=async_main(service_config), connection_limit=target_peer_count)\n', '  File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 323, in async_run\n    return asyncio.run(coro)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run\n    return runner.run(main)\n', '  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run\n    return self._loop.run_until_complete(task)\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete\n    self.run_forever()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 607, in run_forever\n    self._run_once()\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once\n    handle._run()\n', '  File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run\n    self._context.run(self._callback, *self._args)\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1138, in validate_block_batches\n    success, state_change_summary, err = await self.add_block_batch(\n', '  File "/home/hajes/chia-blockchain/chia/full_node/full_node.py", line 1281, in add_block_batch\n    pre_validation_results: List[PreValidationResult] = await self.blockchain.pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/blockchain.py", line 805, in pre_validate_blocks_multiprocessing\n    return await pre_validate_blocks_multiprocessing(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 345, in pre_validate_blocks_multiprocessing\n    asyncio.get_running_loop().run_in_executor(\n', '  File "/usr/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor\n    executor.submit(func, *args), loop=self)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 783, in submit\n    self._start_executor_manager_thread()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 722, in _start_executor_manager_thread\n    self._launch_processes()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 749, in _launch_processes\n    self._spawn_process()\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 759, in _spawn_process\n    p.start()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.11/multiprocessing/context.py", line 281, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n', '  File "/home/hajes/chia-blockchain/chia/consensus/multiprocess_validation.py", line 107, in batch_pre_validate_blocks\n    required_iters, error = validate_finished_header_block(\n', '  File "/home/hajes/chia-blockchain/chia/consensus/block_header_validation.py", line 975, in validate_finished_header_block\n    if not validate_vdf(\n', '  File "/home/hajes/chia-blockchain/chia/types/blockchain_format/vdf.py", line 61, in validate_vdf\n    tb = traceback.format_stack()\n'] INVALID VDF INFO. Have: VDFInfo { challenge: 63965ea158e233000b1a169cbc0c589ae37c089856d396ab14fff9822abbca59, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } } Expected: VDFInfo { challenge: 4ea0308de61a8182cd172ff314f61e3545fb3fd60fd431a6583c76e94e461055, number_of_iterations: 10264189, output: ClassgroupElement { data: 00009d12729abdc3b4a7509ce520cf3bae085cfbeeb26c135f8b15dd7723bcefb7f7db673169359adeaf6ad94f1b63e105546fdea3ca111148efb924a162123a4532914cb4cbbdcc953178b8360eb1a8b0f9598eb07dd63c9e2019f630a43a68a8150100 } }
2024-07-19T23:59:30.999 full_node chia.full_node.full_node: ERROR    Invalid block from peer: PeerInfo(_ip=IPv4Address('114.230.221.69'), _port=8444) Err.INVALID_RC_IP_VDF
2024-07-19T23:59:31.007 full_node chia.full_node.full_node: ERROR    sync from fork point failed: ValueError: Failed to validate block batch 5666278 to 5666309
2024-07-19T23:59:31.022 full_node chia.full_node.full_node: ERROR    failed fetching 5666438 to 5666469 from peers
2024-07-19T23:59:31.437 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
2024-07-20T00:04:40.077 full_node chia.full_node.full_node: ERROR    failed fetching 1920 to 1951 from peers
2024-07-20T00:06:53.376 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
2024-07-20T03:55:33.777 full_node chia.full_node.full_node: ERROR    sync from fork point failed: DatabaseError: database disk image is malformed
2024-07-20T03:56:28.779 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
2024-07-20T06:03:34.261 full_node asyncio                 : ERROR    Exception in callback set_result(<Future pendi...ask_wakeup()]>, None) at /home/hajes/chia-blockchain/venv/lib/python3.11/site-packages/aiosqlite/core.py:40
2024-07-20T08:10:05.340 full_node asyncio                 : ERROR    Task exception was never retrieved
2024-07-20T11:51:34.711 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
2024-07-20T13:11:59.728 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak a8f6dae7b3fdf6b683bff211a4573adcc336d72a0c24f391e33f9c3944fab1dc
2024-07-20T13:19:22.968 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 63e45f57298a868b34469e89267e1e683f72de5cb0d995df7c284d02b3dc9a6b
2024-07-20T23:22:14.582 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T01:38:15.139 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T01:48:31.310 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T01:55:46.183 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T04:45:36.613 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T08:24:52.552 full_node chia.full_node.full_node_api: ERROR    got weight proof request for unknown peak 8e7b663523cecfb5c72679e34b18a721fb4fe8751f89f0960a4fdbbe8a134946
2024-07-21T11:18:54.027 full_node full_node_server        : ERROR    Exception Cannot write to closing transport, exception Stack: Traceback (most recent call last):
2024-07-21T13:09:59.362 full_node full_node_server        : ERROR    Exception: Failed to validate block b'\x1bb\x1f\xaa\xab\x0b\x8fu\x14\xca5Q\x94\xbe\x1a\xfb\xa1B\x82f\xb9\x91|\x8f\xab\xcf\xfcaR\x929#' height 5673985: INVALID_TRANSACTIONS_FILTER_HASH, PeerInfo(_ip=IPv4Address('109.151.186.34'), _port=28744). Traceback (most recent call last):
2024-07-21T13:09:59.364 full_node full_node_server        : ERROR    Exception: Failed to validate block b'\x1bb\x1f\xaa\xab\x0b\x8fu\x14\xca5Q\x94\xbe\x1a\xfb\xa1B\x82f\xb9\x91|\x8f\xab\xcf\xfcaR\x929#' height 5673985: INVALID_TRANSACTIONS_FILTER_HASH <class 'ValueError'>, closing connection PeerInfo(_ip=IPv4Address('109.151.186.34'), _port=28744). Traceback (most recent call last):
hajes commented 1 month ago

impossible to chia start farmer -r anymore

2024-07-21T21:36:18.257 daemon chia.daemon.server : INFO Connection close requested. Closing websocket with ['Unknown']. 2024-07-21T21:37:09.588 daemon chia.daemon.server : INFO Daemon Server stopping, Services stopped: [] 2024-07-21T21:37:09.588 daemon chia.daemon.server : INFO Connection closed. Closing websocket with ['Unknown']. 2024-07-21T21:37:09.589 daemon chia.daemon.server : INFO chia daemon exiting 2024-07-21T21:37:09.589 daemon chia.daemon.server : INFO Daemon WebSocketServer closed 2024-07-21T21:37:12.109 daemon chia.daemon.server : INFO chia-blockchain version: 2.4.2 2024-07-21T21:37:12.136 daemon chia.daemon.server : INFO Starting Daemon Server (localhost:55400) 2024-07-21T21:37:14.162 daemon chia.daemon.server : ERROR problem starting chia_harvester Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1286, in start_service process, pid_path = launch_service(self.root_path, exe_command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1468, in launch_service process = subprocess.Popen( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'chia_harvester' 2024-07-21T21:37:14.164 daemon chia.daemon.server : ERROR problem starting chia_farmer Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1286, in start_service process, pid_path = launch_service(self.root_path, exe_command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1468, in launch_service process = subprocess.Popen( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'chia_farmer' 2024-07-21T21:37:14.166 daemon chia.daemon.server : ERROR problem starting chia_full_node Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1286, in start_service process, pid_path = launch_service(self.root_path, exe_command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1468, in launch_service process = subprocess.Popen( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'chia_full_node' 2024-07-21T21:37:14.168 daemon chia.daemon.server : ERROR problem starting chia_wallet Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1286, in start_service process, pid_path = launch_service(self.root_path, exe_command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/daemon/server.py", line 1468, in launch_service process = subprocess.Popen( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'chia_wallet' 2024-07-21T21:37:14.169 daemon chia.daemon.server : INFO Connection close requested. Closing websocket with ['Unknown'].

hajes commented 1 month ago

looks like latest version doesn't like chia-blockchain/venv/bin/chia start farmer -r. after activating python venv, it goes again.

hajes commented 1 month ago

I have today observed a live crash. Watchdog script kicked in, chia stop all -d successfully executed...yet, the following chia stuff still running. chia start farmer -r also failed. What the hell is going on?

ps aux | grep chia hajes 2113688 0.0 2.1 1326160 354124 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113689 0.0 2.1 1326160 354012 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113690 0.0 2.1 1326160 354584 ? S Jul21 0:18 chia_full_node_block_validation_worker hajes 2113691 0.0 2.1 1326160 354072 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113692 0.0 2.1 1326160 353788 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113693 0.0 2.1 1326160 354188 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113694 0.0 2.1 1326160 353844 ? S Jul21 0:14 chia_full_node_block_validation_worker hajes 2113695 0.0 2.1 1326160 354268 ? S Jul21 0:18 chia_full_node_block_validation_worker hajes 2113696 0.0 2.1 1326160 354144 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113697 0.0 2.1 1326160 354044 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113698 0.0 2.1 1326160 354028 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113699 0.0 2.1 1326160 354524 ? S Jul21 0:18 chia_full_node_block_validation_worker hajes 2113700 0.0 2.1 1326160 354412 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113701 0.0 2.1 1326160 354028 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113703 0.0 2.1 1326160 354456 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113704 0.0 2.1 1326160 354884 ? S Jul21 0:18 chia_full_node_block_validation_worker hajes 2113705 0.0 2.1 1326160 356532 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113706 0.0 2.1 1326160 356116 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113707 0.0 2.1 1326160 353800 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113708 0.0 2.1 1326160 354084 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113709 0.0 2.1 1326160 354616 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113710 0.0 2.1 1326160 354120 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113711 0.0 2.1 1326160 353924 ? S Jul21 0:15 chia_full_node_block_validation_worker hajes 2113712 0.0 2.1 1326160 353968 ? S Jul21 0:14 chia_full_node_block_validation_worker hajes 2113713 0.0 2.1 1326160 354136 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113714 0.0 2.1 1326160 354056 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113715 0.0 2.1 1326160 354008 ? S Jul21 0:13 chia_full_node_block_validation_worker hajes 2113716 0.0 2.1 1326160 354284 ? S Jul21 0:17 chia_full_node_block_validation_worker hajes 2113717 0.0 2.1 1326160 353924 ? S Jul21 0:16 chia_full_node_block_validation_worker hajes 2113718 0.0 2.1 1326160 354060 ? S Jul21 0:13 chia_full_node_block_validation_worker hajes 2113719 0.0 2.1 1326160 354416 ? S Jul21 0:18 chia_full_node_block_validation_worker hajes 2113720 0.0 2.1 1326160 353936 ? S Jul21 0:14 chia_full_node_block_validation_worker hajes 2114674 0.0 2.2 1399972 370480 ? S Jul21 0:12 chia_full_node_mempool_worker hajes 2114675 0.0 2.2 1399972 369632 ? S Jul21 0:12 chia_full_node_mempool_worker hajes 2246173 0.0 0.0 6332 2064 pts/1 S+ 06:41 0:00 grep chia

With following error

Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/util/lock.py", line 42, in acquire self._lock.acquire(timeout=timeout, poll_interval=poll_interval) File "/home/hajes/chia-blockchain/venv/lib/python3.11/site-packages/filelock/_api.py", line 304, in acquire raise Timeout(lock_filename) # noqa: TRY301 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ filelock._error.Timeout: The file lock '/home/hajes/.chia/mainnet/run/full_node.lock' could not be acquired. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 196, in run with Lockfile.create(service_launch_lock_path(self.root_path, self._service_name), timeout=1): File "/home/hajes/chia-blockchain/chia/util/lock.py", line 29, in enter self.acquire(timeout=self.timeout, poll_interval=self.poll_interval) File "/home/hajes/chia-blockchain/chia/util/lock.py", line 44, in acquire raise LockfileError(e) from e chia.util.lock.LockfileError: The file lock '/home/hajes/.chia/mainnet/run/full_node.lock' could not be acquired. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/hajes/chia-blockchain/venv/bin/chia_full_node", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/hajes/chia-blockchain/chia/server/start_full_node.py", line 102, in main return async_run(coro=async_main(service_config), connection_limit=target_peer_count) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 323, in async_run return asyncio.run(coro) ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/home/hajes/chia-blockchain/chia/server/start_full_node.py", line 85, in async_main await service.run() File "/home/hajes/chia-blockchain/chia/server/start_service.py", line 201, in run raise ValueError(f"{self._service_name}: already running") from e ValueError: full_node: already running

Only pkill -9 -f chia worked.

I came back from work, farmer dead again.

BrandtH22 commented 1 month ago

Hey @hajes , it looks like the issues might be related to the db being corrupt:

2024-07-20T03:55:33.777 full_node chia.full_node.full_node: ERROR    sync from fork point failed: DatabaseError: database disk image is malformed

I would:

Once chia is synced you will need to update the config file for any custom settings:

Let us know if you run into any issues or have any questions in the process and keep in mind that we can generally provide more timely and thorough support in our discord server (https://discord.gg/chia)

hajes commented 1 month ago

Thanks for reply @BrandtH22, and my apologies for Discord rage few days ago...one should sleep overnight instead of trying to fix something, and then talk blulsiht... :o)

Already did all above as written in initial post. What you mention is 2 days old...all this crashing corrupted database, and refused to sync.

After fresh install, still crashing all the time. I have log in debug mode. It is almost 300MB compressed if you are interested

link

Last crash was about 7:30, and then somewhere between 7:30 - 16:00 no idea when exactly it crashed when I was at work

So far I have a kludge - watchdog that watches logs, and restart chia or kill ghost processes. So far it seems to work...just 15 hours gone in name of new state-of-art farmer :-(

hajes commented 1 month ago

Reinstalled whole system, and it is running. Something got screwed up during/after install.

hajes commented 1 month ago

My celebration of "success" was premature. Freshly installed farmer crashed after 16 hours. Chia official systemd script didn't restart the failing process.

I noticed chia processes eat lots of memory/swap. 1x14 threads C4 should use <500MB of RAM. System has got 16GB of RAM + 2GB swap. It is filled with buffered crap, memory leaks...no idea.

now what?

wjblanke commented 1 month ago

Through other issues I think the thought on our side is there is some kind of hardware issue causing system flakiness. It may be some other component besides the RMAd motherboard at fault.