sodabrew
commented
8 years ago Do you have reconnect enabled? If not, then the effect of a mysql_ping on a dead connection will be raising an exception.
Open technopoetic opened 8 years ago
sodabrew
commented
8 years ago Do you have reconnect enabled? If not, then the effect of a mysql_ping on a dead connection will be raising an exception.
technopoetic
commented
8 years ago Yes, reconnect is enabled. My working hypothesis is that mysql_ping reconnects a dead connection and somehow that kills the current thread. When passenger sees the thread die, then it raises the ThreadHandler::Interrupted. I guess that makes the question, "What happens to the current thread when mysql_ping reconnects?"
And thanks for responding by the way.
sodabrew
commented
8 years ago A failure to reconnect should ultimately raise a Mysql2::Error rather than just terminating the thread. Is it possible you have contention on the connection, i.e. if the mysql_ping is called before the connection is actually checked out of the pool and locked by the thread?
technopoetic
commented
8 years ago I suppose it's possible. I'll have to look at it with an eye towards contention. I'm wondering about what happens when the reconnect succeeds. Passenger's request handler is where that exception comes from. mysql_ping is where we are when it happens, but this is the code that actually raises the exception:
def terminate_threads
debug("Stopping all threads")
threads = @threads_mutex.synchronize do
@threads.dup
end
threads.each do |thr|
thr.raise(ThreadHandler::Interrupted.new)
end
threads.each do |thr|
thr.join
end
debug("All threads stopped")
endAnd that gets called at the end of the main event loop, so it's almost like Passenger is starting a completely new process every time we get a reset connection.
I know that mysql_ping causes things to happen server side, like rolling back transactions, but does it cause anything to happen client side? Or should we just get a new connection and continue on our merry way?
We've been having this issue in production for quite a while now, most of this year, actually. It typically occurs several times a day, but it seems random. So far I haven't been able to track down what's causing it, but my hypothesis is that when mysql_ping initiates a reconnect, it interrupts the current thread, causing Phusion passenger to raise a FATAL exception. I've also had virtually no luck reproducing it locally. Do you guys have any insight?
The fatal exception is: PhusionPassenger::RequestHandler::ThreadHandler::Interrupted
Stacktrace:
To be honest, I'm not sure if this is an issue with Passenger, ActiveRecord or MySQL2. So I figured I'd start here. Thanks.