Open bmaupin opened 3 years ago
Thanks for the detailed report on the issue, @bmaupin!
I've read through the other issue, and cross-checked with the source code here; It does seem that this package doesn't properly handle connection errors after the initial connection:
normally any time there's a database issue, the application automatically reconnects to the database without any intervention on our part.
Unfortunately I'm not sure why this is the case; This could be due to a retry mechanism either in the mysql package or the transport protocol.
I don't see a single mention of fatal in this library. Maybe some extra logic needs to be added to handle fatal errors?
My understanding is that any time a connection encounters an error, it's supposed to be released from the pool to make room for new connections.
When starting up, the connector does check for any errors, but does not distinguish between fatal and non-fatal:
Though this check only happens during app startup; Afterwards, it does not seem that there's any re-connection logic in either the connector nor the Juggler ORM.
When .query()
is called, the response is always passed to handleResponse
:
Which should throw an error; This should cause the application to continue or crash (depending on several factors), but it shouldn't hang.
NB: The use of setHttpCode
in the above code snippet is a remnant of how LB2 and LB3 works. This isn't as relevant for LB4.
I've not tested for a potential solution, but I suspect the issue is with this line:
This seems to be where all query logic converges at, and there's no check for fatal errors.
Unfortunately, why is hangs is still a mystery.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
We're using loopback-connector-mysql v6.0.2 and normally any time there's a database issue, the application automatically reconnects to the database without any intervention on our part.
However, we recently ran into an issue where this did not happen. Even though the database was running, the application would not reconnect to the database until I restarted it.
These were the errors in the log:
(the last error is repeated, I'm guessing for each connection in the pool).
In searching online for similar errors, using a pool seems to be the solution for this sort of problem. Of course, this connector is already using a pool, so I'm not sure what options I have to fix this. Unfortunately despite my best efforts I've been unable to reproduce that exact error and so I can't provide a way to reproduce it at this time.
My understanding is that any time a connection encounters an error, it's supposed to be released from the pool to make room for new connections. Why didn't that happen?
According to https://github.com/mysqljs/mysql/issues/2522#issuecomment-921877429, I wonder if it could be a bug in this library since we aren't calling any of the underlying methods directly:
I don't see a single mention of
fatal
in this library. Maybe some extra logic needs to be added to handle fatal errors? Even crashing the application would be preferable to it hanging, because at least then we could know to restart it.Thanks!