search

vmware-archive / cfops

This is simply an automation that is based on the supported way to back up Pivotal Cloud Foundry
http://www.cfops.io
Apache License 2.0
35 stars 24 forks source link

Cannot connect to MySQL service after a full restore #80

Closed jtgammon closed 8 years ago

jtgammon commented 8 years ago

After doing a full restore from zero, I cannot connect to a previously working MySQL service. The backups and restore shows success, and everything deploys properly via bosh during the restore. Logs are attached below.

Process I followed:

  1. Deployed spring-music, which successfully connected to a mysql service called "jgdb". Successfully browsed the MySQL DB, edited, etc.
  2. Backed up opsman, ert, and mysql-tile.
  3. Destroyed all the VM's in vcenter (mass delete).
  4. Deployed new Ops Manager.
  5. Restored Ops Manager.
  6. Ops Manager Deploy.
  7. Restored MySQL-tile.
  8. Restored ERT.
  9. Apps started/crashed, started/crashed repeatedly due to inability to connect to MySQL services.

NOTE: If I create a new MySQL DB service after the restore, the app will connect to it no problem. The only issue is connecting to a MySQL DB service that has been restored from backup.

Here are the debug logs for the backup and restore of opsman, ert, mysql: bu-restore-logs.tar.gz

Here is the error log from an app trying to connect to restored DB service: spring-music.log.gz

Note: 2016-04-28T09:10:25.70-0400 [APP/0] OUT 13:10:25,702 WARN JdbcServicesImpl:204 - HHH000342: Could not obtain connection to query metadata : Cannot create PoolableConnectionFactory (Access denied for user 'NZc3xPbc9uRiZ52F'@'10.65.187.86' (using password: YES))

2016-04-28T09:10:26.23-0400 [APP/0] OUT Caused by: java.sql.SQLException: Access denied for user 'NZc3xPbc9uRiZ52F'@'10.65.187.86' (using password: YES)

jtgammon commented 8 years ago

CFOPS 2.2.7 OpsMan 1.6.11 ERT 1.6.20 MySQL 1.7.6

screen shot 2016-04-28 at 9 36 58 am

calebwashburn commented 8 years ago

Do you have a copy of mysql-tile.dmp that can be shared?

Caleb Washburn PCF Advisory Solutions Architect cwashburn@pivotal.io

On Apr 28, 2016, at 6:40 AM, jtgammon notifications@github.com wrote:

CFOPS 2.2.7 OpsMan 1.6.11 ERT 1.6.20 MySQL 1.7.6

https://cloud.githubusercontent.com/assets/3753557/14887669/ca0af774-0d24-11e6-8c60-a1581edb78b1.png — You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/pivotalservices/cfops/issues/80#issuecomment-215425735

jtgammon commented 8 years ago

We got it working by connecting to the DB and doing the "flush privileges" command. After this, the apps start. I believe this procedure is missing from the restore process?

From: https://docs.pivotal.io/p-mysql/backup.html command: $ mysql -u root -p -h $MYSQL_NODE_IP -e "FLUSH PRIVILEGES"

Below is the mysql tile dump: mysql-tile.dmp.gz

calebwashburn commented 8 years ago

@jtgammon - Fixed with version v0.0.19 of mysql-plugin (http://www.cfops.io/plugins/mysql-plugin/)