App losing DB credentials

[jbenezech]

I installed an application based on Dancer using the Dancer::Plugin::Database module at the beginning of the week and notice that the connection to the mysql DB got lost after a few days of running.

Logs report crashed: DBI connect('database=xxxxx','xxxxx',...) failed: Access denied for user 'xxx'@'localhost' (using password: YES)

The authentication method is user/password and they are of course correct. Mysql was fine at the time the application was crashing and I could connect with mysql client using the application credentials. Restarting the application solved the problem. The application is deployed with plask: sudo -u www-data /usr/local/bin/plackup -E production -s Starman --workers=1 -l /tmp/plack.sock -a bin/app.pl & I am using the Dancer::Plugin::Database plugin with connection_check_threshold: 10

What could cause an access denied ? According to mysql doc, this message means wrong password. Could the in-memory password just get lost ? Weird.

[bigpresh]

Sorry for the slow reply here. This is, indeed, weird.

I can't see why it would be "losing" the credentials. I take it the username in the "Access denied for user..." message was the correct username it should have been using?

Do you still have this issue? Was it with the latest Dancer::Plugin::Database version on CPAN?

[jbenezech]

Thanks a lot for your reply. I didn't see that problem again on the application that was deployed at that time and, as I could not reproduce the problem, didn't think much more of it. But recently I deployed another application which was getting more traffic and the problem appeared again. The database connection would fail at some point with the same error. To make things even more weird, when the customer notified me of the problem, the application was up, meaning that somehow the connection issue had resolved itself (nobody had restarted the app).

Now, I think I have solved the issue (hopefully) and this might help narrow down the cause. The connection was over tcp to localhost on standard port. I've switched to unix socket and haven't seen the problem again since.

I'm starting to think that this is not a problem with this plugin but a TCP issue, maybe a timeout, and a mysql driver reporting an incorrect error message.

[bigpresh]

Please do let me know if you see the problem again now you've switched to Unix sockets. I certainly haven't seen this kind of problem in my use of D::P::D in production apps.

If it happens again, maybe a database_connection_failed which dumps out the config to verify that you still have the right username & password stored there would be useful?

If not - probably close this issue, agree?

[jbenezech]

Funny you just replied now. The app went down yesterday again, same access denied error even using sockets. Putting more logging in place and will do the config dump on database_connection_failed.

[bigpresh]

Ah - are you using Dancer::Plugin::EscapeHTML with automatic_escaping? I seem to recall some issues where itcould accidentally munge stuff in the config if the config hashref is passed as a reference to the template params . Or any other similar plugin/code that could be accidentally walking the config and changing things?