search

modxevil / google-cloud-sdk

Automatically exported from code.google.com/p/google-cloud-sdk
0 stars 0 forks source link

Problem creating databases with deployment manager #333

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?

Try creating a deployment with one cloud SQL server and 10 databases. I've 
attached an example configuration file for doing this.

What is the expected output? What do you see instead?

I expect the cloud SQL server and the 10 databases to be created. Instead, I 
get various errors:

The first time the "gcloud deployments create" is run I get errors for all 
databases. I assume this is because gcloud doesn't prooperly wait for the sql 
instance to get running before attempting to create the databases. So I get 
errors like this:

Waiting for create 
operation-1445548182725-522b7e9efe189-816ce681-4324004d...failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation 
operation-1445548182725-522b7e9efe189-816ce681-4324004d: <ErrorValue
 errors: [<ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb5'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb3'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb4'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb10'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb6'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb2'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb9'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb7'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 409 {statusMessage=Conflict, errors=[{message=The instance or operation is not in an appropriate state to handle the request., domain=global, reason=invalidState}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb8'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 409 {statusMessage=Conflict, errors=[{message=The instance or operation is not in an appropriate state to handle the request., domain=global, reason=invalidState}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb1'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 409 {statusMessage=Conflict, errors=[{message=The instance or operation is not in an appropriate state to handle the request., domain=global, reason=invalidState}]}'>]>

If wait a little while, and then run "gcloud deployment-manager deployments 
update", I instead get these errors:

Waiting for update 
operation-1445548878087-522b813624358-4644b69e-b8d8d576...failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation 
operation-1445548878087-522b813624358-4644b69e-b8d8d576: <ErrorValue
 errors: [<ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb7'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb7'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb9'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb9'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb8'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb8'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb4'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb4'; database exists">]>

All databases do appear in the web ui, but gcloud doesn't seem to acknowledge 
this. So, I manually delete the databases testdb4, testdb7, testdb8 and 
testdb9. When I run the update command again, I get this error:

Waiting for update 
operation-1445549207463-522b827042459-d7df34b1-64b2b5b9...failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation 
operation-1445549207463-522b827042459-d7df34b1-64b2b5b9: <ErrorValue
 errors: [<ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'test-deployment/testdb9'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb9'; database exists">]>

So, now it is only testdb9 that fails. I remove it manually (again) and re-run 
the update command again. Now it succeeds.

What is the output of 'gcloud info'?

Google Cloud SDK [0.9.82]

Platform: [Linux, x86_64]
Python Version: [2.7.10 (default, Jan 01 1970, 00:00:01)  [GCC 4.9.3]]
Site Packages: [Disabled]

Installation Root: 
[/nix/store/3fs478kqi88rmrnyaw11da8ad4hxsqi6-google-cloud-sdk-0.9.82/google-clou
d-sdk]
Installed Components:
  core: [2015.10.08]
  core-nix: [2015.10.08]
  gcloud: [2015.10.08]
  gsutil-nix: [4.15]
  gsutil: [4.15]
  bq: [2.0.18]
  bq-nix: [2.0.18]
System PATH: 
[/home/rickard/bin:/var/setuid-wrappers:/home/rickard/.nix-profile/bin:/home/ric
kard/.nix-profile/sbin:/home/rickard/.nix-profile/lib/kde4/libexec:/nix/var/nix/
profiles/default/bin:/nix/var/nix/profiles/default/sbin:/nix/var/nix/profiles/de
fault/lib/kde4/libexec:/run/current-system/sw/bin:/run/current-system/sw/sbin:/r
un/current-system/sw/lib/kde4/libexec]
Cloud SDK on PATH: [False]

Installation Properties: 
[/nix/store/3fs478kqi88rmrnyaw11da8ad4hxsqi6-google-cloud-sdk-0.9.82/google-clou
d-sdk/properties]
User Config Directory: [/home/rickard/.config/gcloud]
User Properties: [/home/rickard/.config/gcloud/properties]
Current Workspace: [None]
Workspace Config Directory: [None]
Workspace Properties: [None]

Account: [xxxx]
Project: [xxxx]

Current Properties:
  [core]
    project: [xxxx]
    account: [xxxx]
  [app]
    use_appengine_api: [True]

Logs Directory: [/home/rickard/.config/gcloud/logs]
Last Log File: 
[/home/rickard/.config/gcloud/logs/2015.10.22/23.26.47.014174.log]

Please provide any additional information below.

Original issue reported on code.google.com by rickyn...@gmail.com on 22 Oct 2015 at 9:41

Attachments:

GoogleCodeExporter commented 8 years ago 
Thanks for reporting this; we're sorry about the delay.

We're looking into it.

Original comment by z...@google.com on 24 Nov 2015 at 10:33

GoogleCodeExporter commented 8 years ago 
A couple of questions to help us further troubleshoot:

Do you still experience the same issue if you try again now? We may be able to 
dig up logs to see what's happening on our end. Please provide the operation 
names if you see this (they're in the gcloud output).

Are you by any chance applying previews between those commands?

Original comment by z...@google.com on 25 Nov 2015 at 2:44

GoogleCodeExporter commented 8 years ago 
Hi,

I have not applied previews between the commands (not sure what you mean by 
that, but I've not done anything but wait a while between the commands).

I retried it with the same version of google-cloud-sdk (0.9.82) and got the 
same errors as before. Then I updated to the latest version of google-cloud-sdk 
(0.9.88) and got the same behavior again.

Below is the output from my latest run of deployment-manager create/update (I 
used the exact same config as attached earlier, except for changing name of 
deployment and db server):

$ gcloud --project xxxx deployment-manager deployments update rn333-4 --config 
config
Waiting for create 
operation-1448572952203-525782c3f4ef8-d2ad7594-c74d0c70...failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation 
operation-1448572952203-525782c3f4ef8-d2ad7594-c74d0c70: <ErrorValue
 errors: [<ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb1'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb8'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb5'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb4'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb7'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb9'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb2'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb3'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb10'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb6'
 message: u'Unexpected response from resource of type sqladmin.v1beta4.database: 403 {statusMessage=Forbidden, errors=[{message=The client is not authorized to make this request., domain=global, reason=notAuthorized}]}'>]>

$ gcloud --project xxxx deployment-manager deployments update rn333-4 --config 
config
Waiting for update 
operation-1448573714236-5257859ab0260-fae34a35-54686614...failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation 
operation-1448573714236-5257859ab0260-fae34a35-54686614: <ErrorValue
 errors: [<ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb4'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb4'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb6'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb6'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb3'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb3'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb2'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb2'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb10'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb10'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb5'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb5'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb7'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb7'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb9'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb9'; database exists">, <ErrorsValueListEntry
 code: u'RESOURCE_ERROR'
 location: u'rn333-4/testdb1'
 message: u"Unexpected response from resource of type sqladmin.v1beta4.database: ERROR_RDBMS speckle::MySQL exception: mysql_query Can't create database 'testdb1'; database exists">]>

Original comment by rickyn...@gmail.com on 26 Nov 2015 at 9:40

GoogleCodeExporter commented 8 years ago 
Thanks for following up; we'll investigate and get back to you soon.

Original comment by z...@google.com on 27 Nov 2015 at 8:32

GoogleCodeExporter commented 8 years ago 
Still looking into this–it's proving to be a little hairy to figure out 
what's going on. Thanks for your patience.

Original comment by z...@google.com on 7 Dec 2015 at 2:45

GoogleCodeExporter commented 8 years ago 
I got some answers from someone on the deployment manager team:

-----------------------------------------------

Turns out the SQL API is different than most cloud APIs in a few ways, and so 
must be used differently in DM as well.

1. sqladmin API does not allow instance names to be re-used, even after 
deletion. This explains the 409 error "The instance or operation is not in an 
appropriate state to handle the request". The instance name (rntest-db-db01) 
needs to be changed with each new deployment.

2. Only one operation can happen on an instance at a time (including db 
creation). To force this dependency chain, add a reference to each database on 
the previously created resource. For the first db, replace the instance 
property with "$(ref.rntest-db-db01.name)"'. For all subsequent datbases, 
replace the instance property with "$(ref.rntest-db)"

I've attached a config that should correct these issues. To use:
$ gcloud  deployment-manager deployments create rntest-db-dbXX --config 
sqladmin.yaml

The jinja file needs to be in the same folder as the yaml file, and the 
deployment name needs to be unique every time (even after deletion) to enforce 
always-unique instance names.

Original comment by z...@google.com on 8 Dec 2015 at 8:46

Attachments:

GoogleCodeExporter commented 8 years ago 
And...quick correction to point 2 above: the chained property should be 
project, not instance. The sqladmin.jinja example does this correctly.

For the first db, chain to the instance:
project: $(ref.rntest-db-db01.project)

For the rest, chain to database n-1 (example for testdb9):
project: $(ref.testdb8.project)

Original comment by z...@google.com on 8 Dec 2015 at 9:03

GoogleCodeExporter commented 8 years ago 
I have verified that the project-chaining approach works. Thanks!

Original comment by rickyn...@gmail.com on 14 Dec 2015 at 11:00