Use Native Google Cloud Storage for Release Blobs

[johnsonj]

This BOSH release uses Google Cloud Storage (GCS) for storing release blobs in S3 compability mode and should be migrated to using native GCS. This enables service account support and better support for large file uploads.

The latest version of bosh2 (2.0.28-cb77557-2017-07-11T23:04:21Z) supports native GCS as a blobstore (see: cloudfoundry/bosh-cli#238).

Migration Plan

1. Ensure project and developers are using the latest bosh2 (>= 2.0.28-cb77557-2017-07-11T23:04:21Z). This is needed for CI pipelines and wherever releases are built.
2. Sync blobs locally with BOSH v2:

   bosh2 sync-blobs

3. Remove object_ids from config/blobs.yml:

   sed -i '/object_id/d' config/blobs.yml

4. Update config/final.yml:

   ---
   final_name: <<unchanged>>
   blobstore:
    provider: gcs
    options:
      bucket_name: <<unchanged>>
      # remove: host, endpoint, use_ssl

5. Update config/private.yml (secrets for developers and CI, do not check in)

   blobstore:
    options:
      json_key: <<service account key>>

   To generate a new service account/key:

   export project_id=Project hosting your GCS bucket, eg my-gcp-project
   export bucket_name=GCS bucket name, eg my-bosh-release-blobs
   
   export service_account_name=${bucket_name}-blobs
   export service_account_email=${service_account_name}@${project_id}.iam.gserviceaccount.com
   credentials_file=$(mktemp)
   
   gcloud config set project ${project_id}
   gcloud iam service-accounts create ${service_account_name} --display-name "BOSH-CLI access for ${bucket_name}"
   gsutil iam ch serviceAccount:${service_account_email}:objectCreator,objectViewer gs://${bucket_name}
   gcloud iam service-accounts keys create ${credentials_file} --iam-account ${service_account_email}
   
   echo "$(cat ${credentials_file})"

6. Re-upload the blobs to confirm everything works and reassign IDs:

   bosh2 upload-blobs