search

cloudfoundry / bosh-google-cpi-release

BOSH Google CPI
Apache License 2.0
63 stars 96 forks source link

oauth2: cannot fetch token when creating VMs #203

Closed alex-slynko closed 6 years ago

alex-slynko commented 7 years ago

We are using BOSH 262 and Google CPI 25.9.0.

Sometimes we have following error when deploying instances

CPI error 'Bosh::Clouds::CloudError' with message 'Setting metadata for vm 'vm-c613e859-e457-42d6-52eb-e3ecba35427b': Failed to find Google Instance 'vm-c613e859-e457-42d6-52eb-e3ecba35427b': Get https://www.googleapis.com/compute/beta/projects/cf-pcf-kubo/aggregated/instances?alt=json&filter=name+eq+.%2Avm-c613e859-e457-42d6-52eb-e3ecba35427b: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: net/http: TLS handshake timeout' in 'set_vm_metadata' CPI method"

There is nothing more in logs about this error.

bstick12 commented 7 years ago

Another failure of a similar type on delete_vm

 L Error: CPI error 'Bosh::Clouds::CloudError' with message 'Deleting vm 'vm-7b119c50-4268-4dff-7877-2817e7f2810e': Failed to find Google Instance 'vm-7b119c50-4268-4dff-7877-2817e7f2810e': Get https://www.googleapis.com/compute/v1/projects/cf-pcf-kubo/aggregated/instances?alt=json&filter=name+eq+.%2Avm-7b119c50-4268-4dff-7877-2817e7f2810e: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 74.125.141.84:443: i/o timeout' in 'delete_vm' CPI method

This one is an i/o timeout.

johnsonj commented 7 years ago

Distinct errors:

Both cases are covered by our current retry logic (playground to verify type switch)

johnsonj commented 6 years ago

Closing as this should be resolved by the cases cited above. Please re-open if this repros.

Also when this does happen it's a good idea to check cloud status. I had some odd timeout errors yesterday that ended up being due to related to an incident.