Missing variables from provided templates

[phong2tran]

We're seeing these errors when validating the manifest ( --var-errs option) for migrating to the cf-deployment:

$ bosh interpolate --var-errs -e am3-test -d cf -v system_domain=am3-test.dctmlabs.com --vars-store=variables/deployment-vars.yml -l variables/static-ip-vars.yml -l variables/ot-vars.yml -o cf-deployment/operations/scale-to-one-az.yml -o cf-deployment/operations/use-haproxy.yml -o cf-deployment-transition/migrate-haproxy.yml -o cf-deployment/operations/legacy/keep-haproxy-ssl-pem.yml -o cf-deployment/operations/use-postgres.yml -o cf-deployment-transition/migrate-postgres.yml -o cf-deployment/operations/legacy/keep-original-postgres-configuration.yml -o cf-deployment-transition/migrate-webdav.yml -o cf-deployment/operations/legacy/keep-original-blobstore-directory-keys.yml -o cf-deployment-transition/keep-etcd-for-transition.yml -o cf-deployment/operations/legacy/keep-static-ips.yml -o cf-deployment/operations/legacy/keep-original-internal-usernames.yml -o operations/scale-bbs-databases.yml -o operations/scale-diego-cells.yml -o operations/scale-routers.yml -o operations/use-specific-stemcell-version.yml -o cf-deployment-transition/cfr-to-cfd.yml -- cf-deployment/cf-deployment.yml

• Expected to find variables:
  • app_package_directory_key
  • bbs_database_name
  • bbs_database_username
  • buildpack_directory_key
  • cc_database_name
  • cc_staging_upload_user
  • droplet_directory_key
  • haproxy_ssl_pem
  • locket_database_name
  • locket_database_username
  • nats_user
  • resource_directory_key
  • router_status_user
  • uaa_database_name
  • uaa_database_username

It looked like we're expected to manually provide these variable values. My question is that most of these values are already available in cf-release/diego manifests, why can we add these variables in the template files (https://github.com/cloudfoundry/cf-deployment-transition/tree/master/templates) and extract these values from the existing manifests as other variables. This would make these values automatically available in the generated deployment-vars.yml when extract-vars-store-from-manifests.sh script is executed.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/155608362

The labels on this github issue will be updated when the story is started.

[dsabeti]

Hey, @phong2tran. Thanks for bringing this up. We actually considered this originally, but it was sort of MVP/nice-to-have distinction that we decided to make. The first few users managed the transition without this feature, so decided not to prioritize it. If you're feeling blocked by this, I'm happy to prioritize the this story as a feature request to add that support.

[wbean1]

Thanks for reporting this, we ran into the same and were slightly confused, esp since we're not really sure how the database migration is going to work (docs don't go into much detail here)... is this asking for existing database_names and database_usernames? Or wanting us to provide new. Agreed its confusing.

[dsabeti]

Hey @wbean1, it looks like we never updated the step-by-step guide with information about this. I'll update that.

To answer your questions directly:

• The migrate-postgres.yml ops-file adds the migrated_from field to the instance group in your manifest. That keyword tells BOSH that, despite the new name for the instance group, it should re-use the same VMs and persistent disks. This should prevent data loss and essentially "migrate" the database. (I put "migrate" in quotes because this "migration," which changes BOSH instance group names, is very different from a normal database migration, which changes database schemas.)
• The parameters for the keep-original-postgres-configuration.yml ops-file in cf-deployment are meant to reflect the same values as you've always used. As mentioned above, we could probably have updated the vars-store extractor to pull those values out of your old cf-release manifest.

[dsabeti]

By the way, the documentation for migrating postgres can be found here: https://github.com/cloudfoundry/cf-deployment-transition/blob/master/transition-options.md#-decide-how-to-handle-your-database

[wbean1]

Thanks @dsabeti for more detail on what the postgres "migration" actually entails; this helped us nail down an issue where if you apply the ops files in the wrong order, like:

(incorrect):

-o ../cf-deployment-transition/migrate-postgres.yml \
-o ../cf-deployment/operations/use-postgres.yml \
-o ../cf-deployment/operations/legacy/keep-original-postgres-configuration.yml \

instead of (correct):

-o ../cf-deployment/operations/use-postgres.yml \
-o ../cf-deployment-transition/migrate-postgres.yml \
-o ../cf-deployment/operations/legacy/keep-original-postgres-configuration.yml \

you end up with a migrated_from of:

  migrated_from:
  - name: postgres
  - name: singleton-database

instead of the desired:

  migrated_from:
  - az: z1
    name: postgres_z1

and the persistent disk isn't reused.

The ordering of the opsfiles isn't called out in the doc, and this is probably just our natural learning about cf-deployment and opsfiles... I can PR some doc changes when we've actually gone through it once to completion.

Thanks again for charting the transition course though; it is much appreciated.

[dsabeti]

Thanks for the update @wbean1. Ordering ops-files can be tricky, and it's a pretty common pain point. I'll surface this feedback to the BOSH team.

A PR to update the docs would be greatly appreciated!

[wbean1]

It also looks like the postgres database (if you're using the internal) doesn't keep its original static_ip config, which is required if your existing diego-release deployment is deployed referencing its ip (I don't think we're unique here, but maybe). We added a keep-database-static-ips ops file; I assume this could be removed once you destroy the diego-release, but we haven't gotten that far yet to confirm.

[dsabeti]

@wbean1 I think we'd want to extend this section to mention databases as well. The idea is that you can edit your cloud-config so that the postgres IP address in included in the dynamic rather than static IP range. So, when you bosh deploy, BOSH will see that a) your postgres IP is in the dynamic range, and b) your postgres job is seeking to be deployed with a dynamic IP (because it no longer contains the static_ips section). Taken together, those two things will lead BOSH to keep your postgres to the same IP address.