Errno::EMFILE exceptions using Moneta with Redis backend

[beet]

Maybe this is more of a question for Moneta, but when deploying CircuitBox to production (ECS containers in AWS), processes started raising Errno::EMFILE exceptions with messages like:

Failed to open TCP connection to [foo.com] (Too many open files - getaddrinfo)

The circuit was configured with:

cache: Moneta.new(:Redis, url: ENV.fetch('REDIS_URL'), driver: 'hiredis')

Was unable to reproduce locally, and haven't ruled out the possibility of it being ECS related, but is there a way to verify that CircuitBox's cache is actually using Redis through Moneta and not the filesystem?

[matthewshafer]

This seems to be a moneta issue but there's something you might want to check. Do you know what the resource limit is set too for the user the process is running under? We've seen issues (non circuitbox related) when we have the open file limit set to a small value.

[beet]

This seems to be a moneta issue but there's something you might want to check. Do you know what the resource limit is set too for the user the process is running under? We've seen issues (non circuitbox related) when we have the open file limit set to a small value.

Good question, will check. cheers

[sideshowcoder]

Remember that TCP streams manifest as fails on unix so this error basically tells you that you can't open the TCP connection because the running user can't open more files. If you can check on the host, with the correct user what ulimit is set to.

[beet]

Thanks, it's an ECS box that we spin up afresh with each deployment:

ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        10240
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) unlimited
process              unlimited
nofiles              1024

Looks like it's set to 1024.

Now I come to think of it, I'm using it to iterate through a collection of records, create a unique circuit breaker for each, and perform an action within it like:

# pseudo code
records.find_each do |record|
  Circuitbox.circuit("record_#{record.id}", { cache: Moneta.new(:Redis) }).run! do
    # potentially dangerous operation
  rescue Circuitbox::OpenCircuitError
    # log circuit breaker opening for this record
  end
end

Could that result in Circuitbox holding a TCP connection to Moneta for each record for the duration of the Ruby process?

[beet]

Confirming that each new Moneta instance does seem to establish a new Redis connection, and can consistently reproduce:

connections = []

2000.times do
  connections << Moneta.new(:Redis, url: ENV.fetch('REDIS_URL'), driver: 'hiredis')
end

connections.each {|connection| connection.key?('foo')}

RuntimeError: System error
from /bundle/ruby/3.0.0/gems/redis-4.2.5/lib/redis/connection/hiredis.rb:20:in `connect'

I think it's catching a lower-level error; attempting to run any system command at this point fails with:

> puts `lsof`
Errno::EMFILE: Too many open files - lsof

So this is user error on my part; I don't think CircuitBox is intended to be used to create an individual circuit for individual DB records, especially not when its interface to the key-value store is through an abstraction layer, which in the case of Moneta + Redis leaves a key for each record with no expiry.