Open yongmingyang opened 8 months ago
+1 this happened with mine too last night.
Hi, can you both please check your open file limits on your systems and report those findings?
For dagger user:
$ ulimit -n
1024
ChatGPT suggested this one:
$ ps -p <wield pid> -o nlwp
NLWP
78
Prob not this one:
$ sudo cat /proc/sys/fs/file-max
9223372036854775807
Open file limits by: ulimit -n : 5000000
2024-03-02T04:03:20.540930133+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283730 finalized! 2024-03-02T04:03:20.541136148+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:20.541359494+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=21 2024-03-02T04:03:20.541435776+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=5 2024-03-02T04:03:20.541525834+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=16 2024-03-02T04:03:20.541605732+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=6 2024-03-02T04:03:20.541686162+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=11 2024-03-02T04:03:20.541771912+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:20.541871237+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:20.541946347+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974899 bundlehash="AymUBYKGSzgFeJRcfBxXUSwGv1DvCZiVRtz9HoCzfkaS" 2024-03-02T04:03:21.096533065+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.098514277+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.098788218+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101181778+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101333110+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101423989+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101551828+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101643088+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101720923+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104661422+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104840326+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104930624+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104959567+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104975067+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.105012817+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.106050110+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.108211177+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.108326402+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.115859209+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283731 finalized! 2024-03-02T04:03:21.116365763+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:21.116586225+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=21 2024-03-02T04:03:21.116680140+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=11 2024-03-02T04:03:21.116766651+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=6 2024-03-02T04:03:21.116840779+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=10 2024-03-02T04:03:21.117077580+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=17 2024-03-02T04:03:21.117228191+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:21.117284465+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=5 2024-03-02T04:03:21.117346842+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974931 bundlehash="8mC5JJU2bLmyeQHXzuSjknWnxuc98XeqadDY3swM9rFm" 2024-03-02T04:03:21.448617694+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.448766632+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.449656440+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.453379227+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.453987912+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454157147+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454322636+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454414367+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454507030+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454598188+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454673299+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454759550+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.455809336+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.463680444+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283732 finalized! 2024-03-02T04:03:21.463980864+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=5 2024-03-02T04:03:21.464326808+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=72 2024-03-02T04:03:21.464414221+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=4 2024-03-02T04:03:21.464569290+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=15 2024-03-02T04:03:21.464712538+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=4 2024-03-02T04:03:21.464780624+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=7 2024-03-02T04:03:21.464919423+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974954 bundlehash="36Q47eBEtxS69yzb4g6B9KqQAwMQFRBKzG1vtLXHacXa" 2024-03-02T04:03:22.063457816+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.063646938+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.064685934+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.066498172+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.066646869+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067478799+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067799256+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067916565+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.068122419+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.069893840+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070138887+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070386949+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070505720+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070623269+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070689061+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.071805262+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.073001991+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.178114171+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.179044906+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.179885152+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.180246545+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.760339163+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.760574192+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.761061480+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.761302759+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.762384184+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.763276356+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.764931992+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765026858+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765159295+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765211823+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.766543886+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.767679642+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.767843347+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.769553404+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.769737377+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770104200+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770231307+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770318149+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771083726+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771230329+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771735831+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.772217148+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.772716378+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.773294465+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.773568526+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.774155760+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.774742263+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.789689057+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283733 finalized! 2024-03-02T04:03:22.789893458+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283734 finalized! 2024-03-02T04:03:22.790014594+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:22.792360195+01:00 [ERROR] log_panics - thread 'wield-metrics-agent' panicked at 'Client::new(): reqwest::Error { kind: Builder, source: Os { code: 24, kind: Uncategorized, message: "Too many open files" } }': /.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.22/src/blocking/client.rs:877 2024-03-02T04:03:22.889573835+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.889706072+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.889719627+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.720772436+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.721898856+01:00 [INFO] dagger::p2p::graph::dagger - Added node to DAGGER in 0 millis; new size is 2200 w/ current nodes 2200 2024-03-02T04:03:23.722658201+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.723784078+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.725241024+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.725332755+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.726129790+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.727163707+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.728484578+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.728683389+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.729776225+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.729960118+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730365193+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730446775+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730483934+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.731469460+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.731970925+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.732395967+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.732884898+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733526885+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733626160+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733667727+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.734529593+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.735451852+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.736718563+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.736798+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.743431772+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283735 finalized! 2024-03-02T04:03:23.743622578+01:00 [ERROR] log_panics - thread 'controller' panicked at 'called' panicked at 'called
Result::unwrap()
on anErr
value: "SendError(..)"': dagger-logger/src/metrics.rs:439 2024-03-02T04:03:23.772064981+01:00 [ERROR] log_panics - thread 'Result::unwrap()
on anErr
value: "SendError(..)"': dagger-logger/src/metrics.rs:439 2024-03-02T04:03:23.996047008+01:00 [ERROR] log_panics - thread 'comms' panicked at 'controller will not drop receiver: SendError { .. }': core/src/p2p/message/handlers/client.rs:434 2024-03-02T04:03:24.435769702+01:00 [ERROR] log_panics - thread 'comms' panicked at 'controller will not drop receiver: SendError { .. }': core/src/p2p/message/handlers/client.rs:434Node just went down completely afterwards