substrakt / letsencrypt-heroku

Make any Heroku application secure in just a couple of minutes.
https://substrakt.com/heroku-ssl-me-weve-come-a-long-way/
MIT License
524 stars 19 forks source link

Redis: Max number of clients reached #15

Closed marcinkoziej closed 7 years ago

marcinkoziej commented 7 years ago

While reloading generation status page i stumbled upon this error:

2016-08-16T14:02:09.261023+00:00 app[web.1]: 2016-08-16 14:02:09 - Redis::CommandError - ERR max number of clients reached:
2016-08-16T14:02:09.261032+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:121:in `call'
2016-08-16T14:02:09.261033+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:102:in `block in connect'
2016-08-16T14:02:09.261033+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:293:in `with_reconnect'
2016-08-16T14:02:09.261034+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:100:in `connect'
2016-08-16T14:02:09.261034+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:364:in `ensure_connected'
2016-08-16T14:02:09.261035+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:221:in `block in process'
2016-08-16T14:02:09.261035+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:306:in `logging'
2016-08-16T14:02:09.261036+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:220:in `process'
2016-08-16T14:02:09.261037+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:120:in `call'
2016-08-16T14:02:09.261037+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:862:in `block in get'
2016-08-16T14:02:09.261038+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:58:in `block in synchronize'
2016-08-16T14:02:09.261038+00:00 app[web.1]:    /app/vendor/ruby-2.3.1/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2016-08-16T14:02:09.261039+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:58:in `synchronize'
2016-08-16T14:02:09.261039+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.1/lib/redis.rb:861:in `get'
2016-08-16T14:02:09.261040+00:00 app[web.1]:    /app/app.rb:22:in `block in <top (required)>'
2016-08-16T14:02:09.261041+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
2016-08-16T14:02:09.261041+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
2016-08-16T14:02:09.261042+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
2016-08-16T14:02:09.261043+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
2016-08-16T14:02:09.261043+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
2016-08-16T14:02:09.261044+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
2016-08-16T14:02:09.261045+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
2016-08-16T14:02:09.261045+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
2016-08-16T14:02:09.261046+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
2016-08-16T14:02:09.261046+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
2016-08-16T14:02:09.261047+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
2016-08-16T14:02:09.261048+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
2016-08-16T14:02:09.261048+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
2016-08-16T14:02:09.261049+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
2016-08-16T14:02:09.261049+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
2016-08-16T14:02:09.261050+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
2016-08-16T14:02:09.261050+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
2016-08-16T14:02:09.261051+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
2016-08-16T14:02:09.261052+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
2016-08-16T14:02:09.261052+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
2016-08-16T14:02:09.261053+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
2016-08-16T14:02:09.261054+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
2016-08-16T14:02:09.261054+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
2016-08-16T14:02:09.261055+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
2016-08-16T14:02:09.261056+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
2016-08-16T14:02:09.261056+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-16T14:02:09.261057+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-16T14:02:09.261058+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
2016-08-16T14:02:09.261058+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
2016-08-16T14:02:09.261063+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
2016-08-16T14:02:09.261064+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'
2016-08-16T14:02:09.261064+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'
2016-08-16T14:02:09.261065+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
2016-08-16T14:02:09.261066+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
2016-08-16T14:02:09.261066+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
2016-08-16T14:02:09.261067+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
2016-08-16T14:02:09.261068+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
2016-08-16T14:02:09.261068+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
2016-08-16T14:02:09.261069+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
2016-08-16T14:02:09.261070+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/configuration.rb:225:in `call'
2016-08-16T14:02:09.261070+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:578:in `handle_request'
2016-08-16T14:02:09.261071+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:415:in `process_client'
2016-08-16T14:02:09.261071+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/server.rb:275:in `block in run'
2016-08-16T14:02:09.261072+00:00 app[web.1]:    /app/vendor/bundle/ruby/2.3.0/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `block in spawn_thread'
maxehmookau commented 7 years ago

Looks like too many clients connecting to Redis. We've already reduced the limit. Out of interest, which Redis provider are you using? Heroku?

marcinkoziej commented 7 years ago

yes, free heroku plan

Marcin Koziej

Akcja Demokracja Ϟ Radiofonia Ϟ Moko Velo Coop GPG key: http://koziej.cahoots.pl/gpg/ Twitter: @movonw

On 08/19/2016 01:45 PM, Max Woolf wrote:

Looks like too many clients connecting to Redis. We've already reduced the limit. Out of interest, which Redis provider are you using? Heroku?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/substrakt/letsencrypt-heroku/issues/15#issuecomment-240997348, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJkNc9oB-N0gi2h1QNrW5FoDkflpsr6ks5qhZdDgaJpZM4Jlb3b.

JaredPon commented 7 years ago
get '/certificate_generation/:token' do
  authenticate!
  content_type :json
  $redis = Redis.new(url: ENV['REDIS_URL'])
  output = {
    token: params[:token],
    status: $redis.get("#{params[:token]}_status"),
    error: $redis.get("#{params[:token]}_error"),
    domain: $redis.get("#{params[:token]}_domain"),
    subdomains: $redis.get("#{params[:token]}_subdomains").split(','),
    message: $redis.get("#{params[:token]}_message")
  }.to_json
  $redis.quit
  output
end

This will fix the issue. A new redis client is being created every time the page is refreshed to check on progress but it isn't being closed.

maxehmookau commented 7 years ago

Perfect @JaredPon. Could you submit a PR with that change please?

avokhmin commented 7 years ago

Maybe better to use Sidekiq.redis{ |conn| conn.get("...") }?

Sidekiq uses connection_pool, see: https://github.com/mperham/sidekiq/blob/v4.1.4/lib/sidekiq/redis_connection.rb

avokhmin commented 7 years ago

Should be enough: https://github.com/substrakt/letsencrypt-heroku/pull/18