GenesysGo / dagger-testnet-issues

A place to report any issues experienced during the DAGGER Testnet waves. No duplicate issues please. Use the search function before submitting an issue. Duplicate issues will be closed.
Apache License 2.0
7 stars 1 forks source link

[ERROR] log_panics & failed to save event in database, #69

Open yongmingyang opened 8 months ago

yongmingyang commented 8 months ago
Screenshot 2024-03-02 at 4 52 59 PM

2024-03-02T04:03:20.540930133+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283730 finalized! 2024-03-02T04:03:20.541136148+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:20.541359494+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=21 2024-03-02T04:03:20.541435776+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=5 2024-03-02T04:03:20.541525834+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=16 2024-03-02T04:03:20.541605732+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=6 2024-03-02T04:03:20.541686162+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=11 2024-03-02T04:03:20.541771912+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:20.541871237+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:20.541946347+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974899 bundlehash="AymUBYKGSzgFeJRcfBxXUSwGv1DvCZiVRtz9HoCzfkaS" 2024-03-02T04:03:21.096533065+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.098514277+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.098788218+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101181778+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101333110+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101423989+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101551828+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101643088+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.101720923+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104661422+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104840326+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104930624+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104959567+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.104975067+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.105012817+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.106050110+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.108211177+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.108326402+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.115859209+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283731 finalized! 2024-03-02T04:03:21.116365763+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:21.116586225+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=21 2024-03-02T04:03:21.116680140+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=11 2024-03-02T04:03:21.116766651+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=6 2024-03-02T04:03:21.116840779+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=10 2024-03-02T04:03:21.117077580+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=17 2024-03-02T04:03:21.117228191+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=12 2024-03-02T04:03:21.117284465+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=5 2024-03-02T04:03:21.117346842+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974931 bundlehash="8mC5JJU2bLmyeQHXzuSjknWnxuc98XeqadDY3swM9rFm" 2024-03-02T04:03:21.448617694+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.448766632+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.449656440+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.453379227+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.453987912+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454157147+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454322636+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454414367+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454507030+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454598188+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454673299+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.454759550+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.455809336+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:21.463680444+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283732 finalized! 2024-03-02T04:03:21.463980864+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=5 2024-03-02T04:03:21.464326808+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=72 2024-03-02T04:03:21.464414221+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=4 2024-03-02T04:03:21.464569290+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=6 block_execution_lapsed_us=15 2024-03-02T04:03:21.464712538+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=3 block_execution_lapsed_us=4 2024-03-02T04:03:21.464780624+01:00 [INFO] dagger_logger::metrics - datapoint: executed_block block_transactions_count=5 block_execution_lapsed_us=7 2024-03-02T04:03:21.464919423+01:00 [INFO] dagger_logger::metrics - datapoint: new_bundlehash new_transaction_count=9974954 bundlehash="36Q47eBEtxS69yzb4g6B9KqQAwMQFRBKzG1vtLXHacXa" 2024-03-02T04:03:22.063457816+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.063646938+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.064685934+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.066498172+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.066646869+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067478799+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067799256+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.067916565+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.068122419+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.069893840+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070138887+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070386949+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070505720+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070623269+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.070689061+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.071805262+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.073001991+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.178114171+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.179044906+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.179885152+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.180246545+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.760339163+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.760574192+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.761061480+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.761302759+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.762384184+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.763276356+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.764931992+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765026858+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765159295+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.765211823+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.766543886+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.767679642+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.767843347+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.769553404+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.769737377+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770104200+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770231307+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.770318149+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771083726+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771230329+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.771735831+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.772217148+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.772716378+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.773294465+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.773568526+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.774155760+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.774742263+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.789689057+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283733 finalized! 2024-03-02T04:03:22.789893458+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283734 finalized! 2024-03-02T04:03:22.790014594+01:00 [INFO] dagger_logger::metrics - datapoint: new_ordered_events new_ordered_events=7 2024-03-02T04:03:22.792360195+01:00 [ERROR] log_panics - thread 'wield-metrics-agent' panicked at 'Client::new(): reqwest::Error { kind: Builder, source: Os { code: 24, kind: Uncategorized, message: "Too many open files" } }': /.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.22/src/blocking/client.rs:877 2024-03-02T04:03:22.889573835+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.889706072+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:22.889719627+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.720772436+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.721898856+01:00 [INFO] dagger::p2p::graph::dagger - Added node to DAGGER in 0 millis; new size is 2200 w/ current nodes 2200 2024-03-02T04:03:23.722658201+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.723784078+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.725241024+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.725332755+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.726129790+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.727163707+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.728484578+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.728683389+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.729776225+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.729960118+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730365193+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730446775+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.730483934+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.731469460+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.731970925+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.732395967+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.732884898+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733526885+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733626160+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.733667727+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.734529593+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.735451852+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.736718563+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.736798+01:00 [ERROR] dagger::p2p::graph::event - Failed to save event in database: IO error: While open a file for random read: /mnt/dag/historydb/086108.sst: Too many open files 2024-03-02T04:03:23.743431772+01:00 [INFO] dagger::p2p::graph::dagger - bundle 283735 finalized! 2024-03-02T04:03:23.743622578+01:00 [ERROR] log_panics - thread 'controller' panicked at 'called Result::unwrap() on an Err value: "SendError(..)"': dagger-logger/src/metrics.rs:439 2024-03-02T04:03:23.772064981+01:00 [ERROR] log_panics - thread '' panicked at 'called Result::unwrap() on an Err value: "SendError(..)"': dagger-logger/src/metrics.rs:439 2024-03-02T04:03:23.996047008+01:00 [ERROR] log_panics - thread 'comms' panicked at 'controller will not drop receiver: SendError { .. }': core/src/p2p/message/handlers/client.rs:434 2024-03-02T04:03:24.435769702+01:00 [ERROR] log_panics - thread 'comms' panicked at 'controller will not drop receiver: SendError { .. }': core/src/p2p/message/handlers/client.rs:434

Node just went down completely afterwards

Aeonia123 commented 8 months ago

+1 this happened with mine too last night.

config.0302.log.gz

tracy-codes commented 8 months ago

Hi, can you both please check your open file limits on your systems and report those findings?

Aeonia123 commented 8 months ago

For dagger user:

$ ulimit -n
1024

ChatGPT suggested this one:

$ ps -p <wield pid> -o nlwp
NLWP
  78

Prob not this one:

$ sudo cat /proc/sys/fs/file-max
9223372036854775807
yongmingyang commented 8 months ago

Open file limits by: ulimit -n : 5000000

Screenshot 2024-03-02 at 10 27 57 PM
g-scooter commented 8 months ago

image