search

paritytech / polkadot-introspector

A collection of tools focused on debugging and monitoring the relay chain and parachain progress from a 🐦-view
GNU General Public License v3.0
37 stars 3 forks source link

Too many open files #552

Open crystalin opened 11 months ago

crystalin commented 11 months ago

Trying to use the kvdb on Moonbeam rocksdb fails:

ERROR polkadot_kvdb] FATAL: cannot start kvdb tool: IO error: While open a file for random read: /var/lib/moonbeam-rocksdb-data/chains/moonbeam/db/full/169392.sst: Too many open files

I'm using default ulimit for Ubuntu 22:

>ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 254990
max locked memory           (kbytes, -l) 8174176
max memory size             (kbytes, -m) unlimited
open files                          (-n) 2048
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 254990
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

There are 20k files in the db/full folder

vstakhov commented 11 months ago

It seems your limit of open files (-n = 2048) is not enough to read that database. I'm not quite sure what is the underlying reason for that large amount of files (I'd check pruning of RocksDB in that case).