"MySQL client is not connected" error in production logs

[NidhiArien]

hi,

We have a multi threaded RAILS 5 consumer which consumes from AWS SQS queues. Lately, with no particular recurrence pattern we see this error in the production log:

"Mysql2::Error: MySQL client is not connected"

From what I have understood so far, that error occurs for two reasons:

1. Connection is closed for some reason on the thread:

https://github.com/brianmario/mysql2/blob/master/spec/mysql2/client_spec.rb#L559

2. When a non standard error is raised:

https://github.com/brianmario/mysql2/blob/master/spec/mysql2/client_spec.rb#L652

The only way I can replicate this issue in non production is by rebooting the RDS instance. So my questions are:

1. Currently I am only logging exception.message to know about the error. I plan to include exception stacktrace as well in the logs to understand better. Is there any other way I can get more information regarding this error?
2. I am using activerecord (4.0.5) and mysql2(0.4.6) in my app. Are there code samples out there which talks about multi-threading using mysql2 gem. Google did not give relevant results for me.

Thank you for your help.

[BoGs]

@NidhiArien possible duplicate of https://github.com/brianmario/mysql2/issues/938? You would need to find the "Lost connection first"

[Jeskz0rd]

Any news about this Statement?

[Dantemss]

I am getting this exact error in Commontator's test suite (I recently added a Travis matrix to test the gem using MySQL as well): https://travis-ci.org/lml/commontator/jobs/549170037

I can reproduce the issue locally using MySQL. PostgreSQL and SQLite3 run the tests just fine both on Travis and locally. Running any individual test in isolation succeeds. Running the entire suite fails on MySQL (almost) every time. So I think a simple app for reproduction is not possible, you'd need enough tests to see the failure. The simplest reproduction steps I can come up with would be:

git clone https://github.com/lml/commontator.git
cd commontator
git checkout aedcb5575b4a2f65d0793c4d3f03b9a6478a2ff1
rbenv/rvm install 2.6.3
rbenv/rvm local/use 2.6.3
bundle install
export COMMONTATOR_DATABASE_ADAPTER=mysql2
export COMMONTATOR_DATABASE_USERNAME=someonewithcreateprivilege
export COMMONTATOR_DATABASE_PASSWORD=passwordfortheabove
rake db:create db:migrate db:seed
rake

config.use_transactional_fixtures = false for rspec-rails solves the issue (but makes the tests slower).

The MySQL error logs contain messages like this (usually 1 per run of the suite but sometimes 2):

2019-06-22T19:03:21.613836Z 216 [Note] Aborted connection 216 to db: 'commontator_test' user: 'commontator' host: 'localhost' (Got an error reading communication packets)

Sometimes it says Got an error writing communication packets instead.

If I catch the first exception with a rescue block, the transactional fixture transaction is left in a state where even a simple command like DummyModel.create returns a NoMethodError for first on NilClass (exec_query is returning nil?) but this may just be a side-effect of this error. If you want to try to catch it anyway, adding a rescue Exception => e with debugger to setup_model_spec in spec/rails_helper.rb should work, but you might have to run rake a few times to get it to catch the error.