Closed djetelina closed 2 months ago
Hi,
Is TLS enabled in the redis instance?
I'm not sure, the chart values in that regard are completely untouched. I would assume it isn't.
Small update, same things happens with external redis. And judging by this, discourse shouldn't use ssl for connecting to redis by default.
One last element, could you try disabling the network policy just to ensure that the timeout issue is not caused by it?
No luck. I guess I wouldn't be able to telnet fromo sidekiq container if that was it, and external redis would work in that case too. It has to be something within the ruby code, that's not playing nice with the environment. But I have absolutely no clue what that could be. Is there a way I could run the rake command with --trace?
You could launch the chart with diagnosticMode.enabled=true, this would override the probes and the command/args so you can enter and perform experiments.
Unfortunately trace doesn't reveal much extra information.
In the container I'm able to run this:
root@test-discourse-6c44c5969f-qd96r:/opt/bitnami/scripts# redis-cli -h xxx -a "<pw>" --verbose ping
PONG
And here's the extra output thanks to --trace
, followed by the TimeoutError you see in my original comment
discourse 07:59:50.30 INFO ==> Populating database
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
Couldn't connect to Redis
rake aborted!
Redis::CannotConnectError: Error connecting to Redis on xxx:6379 (Redis::TimeoutError) (Redis::CannotConnectError)
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/mini_profiler/profiling_methods.rb:89:in `block in profile_method'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `synchronize'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/strings.rb:191:in `get'
/opt/bitnami/discourse/app/models/global_setting.rb:37:in `safe_secret_key_base'
/opt/bitnami/discourse/config/initializers/100-secret_token.rb:6:in `<main>'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:667:in `load'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:667:in `block in load_config_initializer'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:208:in `instrument'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:666:in `load_config_initializer'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:620:in `block (2 levels) in <class:Engine>'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:619:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:619:in `block in <class:Engine>'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `instance_exec'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `run'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:228:in `block in tsort_each'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:50:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:415:in `call'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:347:in `each'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:347:in `call'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:226:in `tsort_each'
/opt/bitnami/ruby/lib/ruby/3.2.0/tsort.rb:205:in `tsort_each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/initializable.rb:60:in `run_initializers'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:372:in `initialize!'
/opt/bitnami/discourse/config/environment.rb:7:in `<main>'
<internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:348:in `require_environment!'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:506:in `block in run_tasks_blocks'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:241:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:241:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/bin/rake:25:in `load'
/opt/bitnami/discourse/vendor/bundle/ruby/3.2.0/bin/rake:25:in `<top (required)>'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli/exec.rb:58:in `load'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli/exec.rb:23:in `run'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli.rb:455:in `exec'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli.rb:35:in `dispatch'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/cli.rb:29:in `start'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/exe/bundle:28:in `block in <top (required)>'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/bitnami/ruby/lib/ruby/gems/3.2.0/gems/bundler-2.5.10/exe/bundle:20:in `<top (required)>'
/opt/bitnami/ruby/bin/bundle:25:in `load'
/opt/bitnami/ruby/bin/bundle:25:in `<main>'
My colleague figured out the issue. It comes from here: https://github.com/bitnami/containers/pull/4174. Where the default is set to no
instead of false
and ruby is then interpreting it as true
. So the following snippet in values fixes it:
discourse:
extraEnvVars:
- name: DISCOURSE_REDIS_USE_SSL
value: "false"
sidekiq:
extraEnvVars:
- name: DISCOURSE_REDIS_USE_SSL
value: "false"
Or at least we're now at a different non-related error, if I get past that and get no more issues, I'll report back and likely submit a PR for upstream fix :)
Thank you so much for letting us know!
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.
Name and Version
bitnami/discourse 13.2.6
What architecture are you using?
None
What steps will reproduce the bug?
Are you using any custom parameters or values?
What is the expected behavior?
Discourse gets initialized
What do you see instead?
Additional information
I tried connecting to the redis through the sidekiq container (telnet) and it works. So does port forwarding the redis service. So the issue must be somewhere else but I just can't figure out what could be causing the read would block error