search

vmware-archive / terraforming-gcp

use terraform, deploy yourself a pcf
Apache License 2.0
71 stars 87 forks source link

internetless gives bosh director an external IP which breaks private google access #107

Closed matthewfischer closed 5 years ago

matthewfischer commented 5 years ago

If you setup with internetless=true your deploy will fail trying to talk to google services (for example):

Task 16 | 21:30:33 | Update stemcell: Uploading stemcell bosh-google-kvm-ubuntu-xenial-go_agent/97.28 to the cloud (cpi: e680d34ef9b5affdd13c) (00:00:30)
                  L Error: CPI error 'Bosh::Clouds::CloudError' with message 'Creating stemcell: Creating Google Image from URL: Failed to create Google Image: Post https://www.googleapis.com/compute/v1/projects/me/global/images?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 209.85.145.84:443: i/o timeout' in 'create_stemcell' CPI method (CPI request ID: 'cpi-630257')
Task 16 | 21:31:03 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'Creating stemcell: Creating Google Image from URL: Failed to create Google Image: Post https://www.googleapis.com/compute/v1/projects/me/global/images?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 209.85.145.84:443: i/o timeout' in 'create_stemcell' CPI method (CPI request ID: 'cpi-630257')

This I think is because we've blocked all egress to 0.0.0.0/0 and given it a public IP. Google Private access does not work with a public IP.

screen shot 2018-11-03 at 3 36 47 pm screen shot 2018-11-03 at 3 39 32 pm
cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

cdutra commented 5 years ago

That's odd. We set destination target to be the service account: https://github.com/pivotal-cf/terraforming-gcp/blob/b99033bce0a62a84a82c3226b980bca2290ecd5c/terraforming-pas/main.tf#L138 Are you configuring the Ops Manager with the output opsman_service_account or selecting the Instance profile?

cdutra commented 5 years ago

To remove the BOSH Director public IP I believe you have to configure Resource config in Ops Manager and uncheck the internet box.

matthewfischer commented 5 years ago

I think I did the service account wrong, I used the one that I passed into terraform. I will retest and let you know.

matthewfischer commented 5 years ago

Yep this was my fault.

jasonbisson commented 5 years ago

I ran into this issue too and it makes sense to use the ops manager service account. Can we update the Internetless readme section to call out that "The Ops Manager VM Service Account" is required for internetess.