Closed FGRibreau closed 11 years ago
I was able to reproduce the error on my laptop with a lot of message that creates a lot of new filters so of course it reached the "too many open files" nearly instantly.
[bloomd]
tcp_port = 8673
udp_port = 8674
data_dir = /data/bloomd
log_level = INFO
cold_interval = 1800
flush_interval = 300
initial_capacity = 10001
default_probability = 0.0001
workers = 4
use_mmap = 1
sudo lsof -p `pidof bloomd` | grep "/data/" | wc -l
472
$ulimit -Hn
unlimited
$ulimit -Sn
2560
Apr 8 15:57:07 MBP.local bloomd[77074] <Critical>: Failed to create new file: /data/bloomd/bloomd.f1.2012-10-2/data.000.mmap for filter f1.2012-10-2. Err: Too many open files
Apr 8 15:57:07 MBP.local bloomd[77074] <Error>: Failed to create SBF: f1.2012-10-2. Err: -24
Apr 8 15:57:07 MBP.local bloomd[77074] <Error>: Failed to fault in the filter 'f1.2012-10-2'. Err: -24
Do I have other choice than incrementing ulimit?
The underlying issue here is that bloomd maintains open file handles for all the filters that are currently in memory. What is happening is that you are eventually just hitting the limit of file handles, so when you try to create a new filter, bloomd returns an internal error.
When bloomd is restarted, it starts without opening the on disk files, and only loads them when read, so it allows more filters to be made for a while.
The simplest solution is to just raise the ulimit values to something huge, that you are unlikely to ever hit. Otherwise, it would require a very large change to bloomd to actively manage the open file handles and work around the limits.
I hope that helps!
Ok! I had the same conclusion as you do, thank you for your time!
I just got an
pybloomd.BloomdError: Got response: Internal Error
(fromfilters[fname] = client.create_filter(fname)
see code below).I have 6 python bloomd clients that send a lot of requests to it. The data dir of bloomd currently contains 1023 bloom filters and counting.
My bloomd configuration is:
Each client, does the following:
with the following python code:
Multiple clients got the
pybloomd.BloomdError: Got response: Internal Error
error within ~50 seconds interval while callingclient.create_filter(fname)
. Some of them were just under huge load so I'm sure they were creating a lot of bloom filters at the same time.It's too bad that this error did not say what was the cause so I've got several questions and feedbacks:
[Update] In fact I've seen this error in the log:
After a restart:
So currently the only way to fix this is to restart bloomd, so I think there may be an issue (or a place for improvement) with the file handling in bloomd.
Again, thanks for your hard work!