search

sigp / lighthouse

Ethereum consensus client in Rust
https://lighthouse.sigmaprime.io/
Apache License 2.0
2.94k stars 745 forks source link

Disk usage spiked with Panic log #5373

Open kim201212 opened 8 months ago

kim201212 commented 8 months ago

Description

Please provide a brief description of the issue.

Version

Please provide your Lighthouse and Rust version. Are you building from stable or unstable, which commit?

Present Behaviour

Describe the present behaviour of the application, with regards to this issue.

Mar 07 21:52:05.093 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  11: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  12: <unknown>
  13: __clone
, slog-async failed with panic: Any { .. }
message: slog::Fuse Drain: Os { code: 28, kind: StorageFull, message: "No space left on device" }, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-2.7.0/src/lib.rs:1944:33
Mar 07 21:52:12.701 INFO New block received                      root: 0x254696066d107df706c14153111d46e4ef94e30332fbf4b9bb4a5759316c701d, slot: 8585359
Mar 07 21:52:13.161 ERRO Database write failed                   action: reverting blob DB changes, error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, service: freezer_db
Mar 07 21:52:13.161 ERRO Database write failed!                  error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, msg: Restoring fork choice from disk, service: beacon
image

Expected Behaviour

How should the application behave?

Steps to resolve

Please describe the steps required to resolve this issue, if known.

jimmygchen commented 8 months ago

Hi @kim201212 Looking at the timing you posted above, the disk was likely already out of space when Lighthouse received the block at 21:52. Could it be other applications / OS on the same machine?

kim201212 commented 8 months ago

Hi @jimmygchen That's the disk where I mounted the volume separately for CL on Ubuntu 20.04.4 LTS. That disk is being used exclusively for the lighthouse BN. I don't think there is any possibility of other applications using it.

Leave a more detailed log here.


Mar 07 21:48:37.261 INFO New block received                      root: 0xcdf219b718a3e2f6a1019cd21a861ff1fa009fc77a99f642ffdd42eff2e3fd72, slot: 8585341
Mar 07 21:48:41.001 INFO Synced                                  slot: 8585341, block: 0xcdf2…fd72, epoch: 268291, finalized_epoch: 268289, finalized_root: 0x471d…3ef6, exec_hash: 0x68c3…b287 (verified), peers: 80, service: slot_notifier
Mar 07 21:48:41.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:48:48.753 INFO New block received                      root: 0x2fb26347ab8a56415ddf9a83376a685cfa386d119b54535d281bc82345efba4b, slot: 8585342
Mar 07 21:48:53.001 INFO Synced                                  slot: 8585342, block: 0x2fb2…ba4b, epoch: 268291, finalized_epoch: 268289, finalized_root: 0x471d…3ef6, exec_hash: 0x21cf…4803 (verified), peers: 83, service: slot_notifier
Mar 07 21:48:53.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:49:01.779 INFO New block received                      root: 0xc575a2002e2ecd5774d17bc1d064572f9fe61c087999e8520e276a680f827a80, slot: 8585343
Mar 07 21:49:05.001 INFO Synced                                  slot: 8585343, block: 0xc575…7a80, epoch: 268291, finalized_epoch: 268289, finalized_root: 0x471d…3ef6, exec_hash: 0x2c79…8b20 (verified), peers: 83, service: slot_notifier
Mar 07 21:49:05.003 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:49:08.473 INFO Current epoch sync signatures           validator: 328286, epoch: 268291, expected: 32, included: 28, service: val_mon, service: beacon
Mar 07 21:49:08.475 INFO Previous epoch attestation(s) success   validators: [<Validators Index>], epoch: 268290, service: val_mon, service: beacon
Mar 07 21:49:11.039 INFO Forwarding register validator request to connected builder, count: 500
Mar 07 21:49:12.077 INFO Forwarding register validator request to connected builder, count: 500
Mar 07 21:49:13.238 INFO New block received                      root: 0x02ad783407133ce17743e0e2891aaf070accdb5f793d16c9d67e9938fa4692b2, slot: 8585344
Mar 07 21:49:17.931 INFO Synced                                  slot: 8585344, block: 0x02ad…92b2, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xd370…b09a (verified), peers: 82, service: slot_notifier
Mar 07 21:49:17.935 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:49:24.948 INFO New block received                      root: 0x666d1b870d832b16e3418a7bb59867a0cd44e2c0fd3ba5a83451a178faab52d0, slot: 8585345
Mar 07 21:49:29.001 INFO Synced                                  slot: 8585345, block: 0x666d…52d0, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x13b4…ed95 (verified), peers: 84, service: slot_notifier
Mar 07 21:49:29.003 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:49:37.450 INFO New block received                      root: 0xe6dc6aece5e79e4a4ec920fe16b273f17cc61b1a37fd1c9ab1f285b10ad226f9, slot: 8585346
Mar 07 21:49:41.001 INFO Synced                                  slot: 8585346, block: 0xe6dc…26f9, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x8eff…5527 (verified), peers: 82, service: slot_notifier
Mar 07 21:49:41.002 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:49:49.651 INFO New block received                      root: 0xbbb07603390eddd025185865ac0bd5cc3eca12d5b13dcdfc69175d626e5fa6bb, slot: 8585347
Mar 07 21:49:53.000 INFO Synced                                  slot: 8585347, block: 0xbbb0…a6bb, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x3929…d3b4 (verified), peers: 84, service: slot_notifier
Mar 07 21:49:53.002 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:50:00.464 INFO New block received                      root: 0x7f09c106b66d89b4c3cb603c5a97658915a0c5d04bef32bb746ea935e44394a3, slot: 8585348
Mar 07 21:50:05.002 INFO Synced                                  slot: 8585348, block: 0x7f09…94a3, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x8254…4fba (verified), peers: 86, service: slot_notifier
Mar 07 21:50:05.004 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:50:13.225 INFO New block received                      root: 0xbc754f6df760615ff39418740a14af1b53cf05e4d2f01137e2531dce7b2ea472, slot: 8585349
Mar 07 21:50:17.001 INFO Synced                                  slot: 8585349, block: 0xbc75…a472, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xd93a…a6c6 (verified), peers: 83, service: slot_notifier
Mar 07 21:50:17.003 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:50:24.305 INFO New block received                      root: 0x093fab8585fafb1d74c443953d990ec745029bea1398448e4ba9d7760c058089, slot: 8585350
Mar 07 21:50:29.001 INFO Synced                                  slot: 8585350, block: 0x093f…8089, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x7464…3a02 (verified), peers: 84, service: slot_notifier
Mar 07 21:50:29.002 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:50:36.316 INFO New block received                      root: 0x6e725b15844797a8603e9391076f0552c68d2900245cd95c678d9da45e4c3781, slot: 8585351
Mar 07 21:50:41.001 INFO Synced                                  slot: 8585351, block: 0x6e72…3781, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xc221…8f1b (verified), peers: 81, service: slot_notifier
Mar 07 21:50:41.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:50:49.061 INFO New block received                      root: 0xcbb1fb977091321b5e682d7d5e121897e6fcc337cc922588480a6bca572b7400, slot: 8585352
Mar 07 21:50:53.032 INFO Synced                                  slot: 8585352, block: 0xcbb1…7400, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x996f…bddd (verified), peers: 83, service: slot_notifier
Mar 07 21:50:53.032 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:51:00.472 INFO New block received                      root: 0x289d5d29ea94e8880c3d4c705778554dc665e0358a2e8b6587dd0e8488e0ab04, slot: 8585353
Mar 07 21:51:05.001 INFO Synced                                  slot: 8585353, block: 0x289d…ab04, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x8c32…9902 (verified), peers: 85, service: slot_notifier
Mar 07 21:51:05.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:51:12.143 INFO New block received                      root: 0xa273a4d331019f461d41d2a23e1965725dcf42a69a7df844350124c6a5aec7b2, slot: 8585354
Mar 07 21:51:17.001 INFO Synced                                  slot: 8585354, block: 0xa273…c7b2, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x589c…7d76 (verified), peers: 83, service: slot_notifier
Mar 07 21:51:17.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:51:24.286 INFO New block received                      root: 0xdeefb06080fbe699d183d5cd6b73600e4cfb6a621d22fc6281f6dbfbac64252f, slot: 8585355
Mar 07 21:51:29.001 INFO Synced                                  slot: 8585355, block: 0xdeef…252f, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x958e…2d75 (verified), peers: 84, service: slot_notifier
Mar 07 21:51:29.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:51:33.691 INFO Imported deposit log(s)                 new: 1, total: 1330691, latest_block: 19384554, service: deposit_contract_rpc
Mar 07 21:51:37.831 INFO New block received                      root: 0xe59fa4bf83c41382c394592fa83632587a376d4f45d95ca706b43b7eebf1cde9, slot: 8585356
Mar 07 21:51:41.001 INFO Synced                                  slot: 8585356, block: 0xe59f…cde9, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0x02f1…c5b4 (verified), peers: 82, service: slot_notifier
Mar 07 21:51:41.005 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:51:48.930 INFO New block received                      root: 0x74b3f0e7bf8c09233c013c25d4f2620670e9efc16463b5f516d71ba36803991c, slot: 8585357
Mar 07 21:51:53.001 INFO Synced                                  slot: 8585357, block: 0x74b3…991c, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xd3e2…d695 (verified), peers: 82, service: slot_notifier
Mar 07 21:51:53.002 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:52:00.681 INFO New block received                      root: 0x73d4e68adc490d1be7b1b3d50f891bcd7ff19ddbcd1d827d6c89db3547c19439, slot: 8585358
Mar 07 21:52:05.001 INFO Synced                                  slot: 8585358, block: 0x73d4…9439, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xedd7…c9fd (verified), peers: 82, service: slot_notifier
Mar 07 21:52:05.003 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:52:05.093 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  11: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  12: <unknown>
  13: __clone
, slog-async failed with panic: Any { .. }
message: slog::Fuse Drain: Os { code: 28, kind: StorageFull, message: "No space left on device" }, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-2.7.0/src/lib.rs:1944:33
Mar 07 21:52:12.701 INFO New block received                      root: 0x254696066d107df706c14153111d46e4ef94e30332fbf4b9bb4a5759316c701d, slot: 8585359
Mar 07 21:52:13.161 ERRO Database write failed                   action: reverting blob DB changes, error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, service: freezer_db
Mar 07 21:52:13.161 ERRO Database write failed!                  error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, msg: Restoring fork choice from disk, service: beacon
Mar 07 21:52:16.643 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642157074s
Mar 07 21:52:16.643 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642156901s
Mar 07 21:52:16.643 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642071833s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642131403s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642350509s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642014345s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642053721s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642197087s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642593975s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642453569s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642433204s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642373654s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642783841s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/beacon/blocks/head/root, status: 500 Internal Server Error, elapsed: 1.643603161s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642705096s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642792878s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642740122s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642753748s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642802296s
Mar 07 21:52:16.644 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.642798409s
Mar 07 21:52:16.645 WARN Error processing HTTP API request       method: GET, path: /eth/v1/validator/attestation_data, status: 500 Internal Server Error, elapsed: 1.643568092s
Mar 07 21:52:16.772 CRIT Beacon block processing error           error: DBError(DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }), service: beacon
Mar 07 21:52:17.001 INFO Synced                                  slot: 8585359, block:    …  empty, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xd370…b09a (verified), peers: 86, service: slot_notifier
Mar 07 21:52:17.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:52:29.001 INFO Synced                                  slot: 8585360, block:    …  empty, epoch: 268292, finalized_epoch: 268290, finalized_root: 0x281c…9f1e, exec_hash: 0xd370…b09a (verified), peers: 86, service: slot_notifier
Mar 07 21:52:29.001 INFO Ready for Deneb                         info: ensure the execution endpoint is updated to the latest Deneb/Cancun release, service: slot_notifier
Mar 07 21:52:31.933 ERRO Database write failed                   action: reverting blob DB changes, error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, service: freezer_db
Mar 07 21:52:31.933 ERRO Database write failed!                  error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }, msg: Restoring fork choice from disk, service: beacon
Mar 07 21:52:33.936 CRIT Beacon block processing error           error: DBError(DBError { message: "Error { message: \"IO error: /root/.lighthouse/data/beacon/chain_db/35110807.log: No space left on device\" }" }), service: beacon

It starts to increase quickly at 21:49 (Epoch 268291 -> 268292). I think panic error is not the root cause.