Move Heroku settings management to Concourse

[blarghmatey]

Right now we are using SaltStack as the mechanism for populating the configuration of our applications in Heroku. This has proven problematic due to the lack of visibility when there are failures to execute the schedule. To resolve this situation we will provision static IAM credentials for use with AWS resources, as well as migrating our population of Heroku settings into Concourse pipelines that are easier for developers to trigger independently and make failures more visible.

### Tasks
- [x] Set up Heroku configurator Python project (repo, poetry, dependencies)
- [x] Identify all secrets currently living in Heroku app configs
- [x] Write Heroku configurator as a Concourse resource that reads configuration vars and values from YAML
- [x] Write Pydantic model that can import the Concourse Vault resource secrets.json and export the herokuconfigurator YAML
- [ ] Write Vault Pydantic model so our pipeline definition can be in Python
- [ ] Figure out the source of truth for configurables created by Pulumi e.g. DATABASE_URL - Salt uses Boto
- [ ] How does the Micromasters Pydantic model integrate with the pipeline?
- [ ] Figure out how to get Pydantic models to populate per environment
- [ ] Write Concourse pipeline definitions to pull Heroku configs from Vault and possibly Consul and write the YAML file the Heroku Configurator Concourse resource defined in the prior step will use
- [ ] Test and remove current Heroku configuration from Saltstack

[blarghmatey]

This is blocked waiting for completion of #1500

[blarghmatey]

The work on #1500 is now complete enough to unblock this work.

[feoh]

Potentially use this Python wrapper to call the Heroku API.

[feoh]

This is an example of the current configuration Salt uses, in this case for the micromasters app.

My tool should suck equivalent configuration out of ol_infrastructure and make the relevant changes in Heroku's config.

[blarghmatey]

For reference here is a sample pipeline for interacting with Vault and loading the values as pipeline vars

resource_types:
- name: vault
  type: docker-image
  source:
    repository: mitodl/concourse-vault-resource
    tag: latest

resources:
- name: vault-postgres-mitxonline
  type: vault
  source:
    address: https://vault-qa.odl.mit.edu:443
    auth_engine: aws
    aws_vault_role: concourse-vault-resource
    secret:
      mount: postgres-mitxonline
      path: readonly
      engine: database

- name: vault-secret
  type: vault
  source:
    address: https://vault-qa.odl.mit.edu:443
    auth_engine: aws
    aws_vault_role: concourse-vault-resource
    secret:
      mount: secret-airbyte
      path: pomerium
      engine: kv2

- name: vault-out
  type: vault
  source:
    address: https://vault-qa.odl.mit.edu:443
    auth_engine: aws
    aws_vault_role: concourse-vault-resource

jobs:
- name: do-something
  plan:
  - get: vault-postgres-mitxonline
    # params:
    #   postgres-mitxonline:
    #     paths:
    #       - readonly
    #     engine: database
  - get: vault-secret
    # params:
    #   secret-airbyte:
    #     paths:
    #       - pomerium
    #     engine: kv2
  - load_var: postgres
    file: vault-postgres-mitxonline/vault.json
  - load_var: secret
    file: vault-secret/vault.json
  - put: vault-out
    no_get: true
    params:
      secret:
        engine: kv1
        secrets:
          test_db:
            db_user: ((.:postgres.postgres-mitxonline-readonly.username))
            db_pass: ((.:postgres.postgres-mitxonline-readonly.password))
          test_pomerium:
            pomerium: ((.:secret.secret-airbyte-pomerium))

[feoh]

Woof.

Just audited the secrets for micromasters that we're currently setting in Heroku with salt, and it's easily 99% of the lines in micromasters.sls.

Makes me wonder if a custom script is even necessary since there should likely be zero logic involved. It'll be a straight "Set this Heroku config value to the value specified in your argument" since we'll be pulling the actual secrets from Vault using Concourse.

The listing is pretty large:

    AWS_ACCESS_KEY_ID:  __vault__:cache:aws-mitx/creds/mitxonline>data>access_key
    AWS_SECRET_ACCESS_KEY: __vault__:cache:aws-mitx/creds/mitxonline>data>secret_key
    {% set rds_endpoint = salt.boto_rds.get_endpoint('mitxonline-{}-app-db'.format(env_data.env_stage)) %}
    {% set pg_creds = salt.vault.cached_read('postgres-mitxonline/creds/app', cache_prefix='heroku-mitxonline') %}
    DATABASE_URL: postgres://{{ pg_creds.data.username }}:{{ pg_creds.data.password }}@{{ rds_endpoint }}/mitxonline
    MAILGUN_KEY: __vault__::secret-operations/global/mailgun-api-key>data>value
    MITOL_GOOGLE_SHEETS_DRIVE_CLIENT_ID: __vault__::secret-mitxonline/google-sheets-refunds>data>drive-client-id
    MITOL_GOOGLE_SHEETS_DRIVE_CLIENT_SECRET: __vault__::secret-mitxonline/google-sheets-refunds>data>drive-client-secret
    MITOL_GOOGLE_SHEETS_DRIVE_API_PROJECT_ID: __vault__::secret-mitxonline/google-sheets-refunds>data>drive-api-project-id
    MITOL_GOOGLE_SHEETS_ENROLLMENT_CHANGE_SHEET_ID: __vault__::secret-mitxonline/google-sheets-refunds>data>enrollment-change-sheet-id
    MITOL_HUBSPOT_API_PRIVATE_TOKEN: __vault__::secret-{{ business_unit }}/hubspot-api-private-token>data>value
    MITOL_HUBSPOT_API_ID_PREFIX: {{ env_data.HUBSPOT_ID_PREFIX }}
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_ACCESS_KEY: __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>access-key
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_PROFILE_ID: __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>profile-id
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_SECURITY_KEY:  __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>security-key
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_MERCHANT_ID: __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>merchant-id
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_MERCHANT_SECRET: __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>merchant-secret
    MITOL_PAYMENT_GATEWAY_CYBERSOURCE_MERCHANT_SECRET_KEY_ID: __vault__::secret-{{ business_unit }}/{{ env_data.env_name }}/cybersource-credentials>data>merchant-secret-key-id
    MITX_ONLINE_REFINE_OIDC_CONFIG_CLIENT_ID: __vault__::secret-mitxonline/refine-oidc>data>client-id
    MITX_ONLINE_REGISTRATION_ACCESS_TOKEN:  __vault__:gen_if_missing:secret-{{ business_unit }}/{{ env_data.openedx_environment }}/mitxonline-registration-access-token>data>value
    OIDC_RSA_PRIVATE_KEY: __vault__::secret-mitxonline/refine-oidc>data>rsa-private-key
    OPEN_EXCHANGE_RATES_APP_ID: __vault__::secret-mitxonline/open-exchange-rates>data>app_id
    OPENEDX_API_CLIENT_ID: __vault__::secret-{{ business_unit }}/{{ environment }}/openedx-api-client>data>client-id
    OPENEDX_API_CLIENT_SECRET: __vault__::secret-{{ business_unit }}/{{ environment }}/openedx-api-client>data>client-secret
    OPENEDX_API_KEY: __vault__:gen_if_missing:secret-{{ business_unit }}/{{ env_data.openedx_environment }}/edx-api-key>data>value
    OPENEDX_SERVICE_WORKER_API_TOKEN: __vault__::secret-{{ business_unit }}/{{ environment }}/openedx-service-worker-api-token>data>value
    RECAPTCHA_SITE_KEY: __vault__::secret-mitxonline/recaptcha-keys>data>site_key
    RECAPTCHA_SECRET_KEY: __vault__::secret-mitxonline/recaptcha-keys>data>secret_key
    SECRET_KEY: __vault__:gen_if_missing:secret-{{ business_unit }}/{{ environment }}/django-secret-key>data>value
    SENTRY_DSN: __vault__::secret-operations/global/mitxonline/sentry-dsn>data>value
    STATUS_TOKEN: __vault__:gen_if_missing:secret-{{ business_unit }}/{{ environment }}/django-status-token>data>value

# Production Only
    HIREFIRE_TOKEN: __vault__::secret-{{ business_unit }}/production-apps/hirefire_token>data>value

[feoh]

After a good conversation with @blarghmatey we're switching gears slightly in that rather than creating a CLI script with a docker container wrapping it to be used within Concourse, we'll actually create a Concourse resource instead which should be cleaner and easier all around.

[feoh]

I'm getting a bit lost among all the moving parts here, so we have:

• A Concourse Vault resource + Pydantic model to enable it to be used in Python wrapped Concourse pipelines
• A Herokuconfigurator Concourse resource + pydantic model
• A per-application pipeline that:
  • Uses the Vault resource to fetch all required Vault secrets for the application being configured
  • Runs some inline Python to process the vault secrets JSON into a YAML file the Heroku configurator can ingest as an input in Concourse
  • Removes the vault.json secrets file from the previous process and fetches the Heroku API Key, thus creating a new one
  • Uses loadvar to set HEROKU_API_KEY based on the previously fetched value
  • Calls the Heroku Configurator resource which ingests the YAML produced by the inline Python and sets the desired Heroku config env variables in the desired Heroku app.

[feoh]

I'm going down the road of just having the Pydantic data model read secrets from Vault and static configs from Consul directly.

To that end, I found a Vault Python API: https://github.com/hvac/hvac

And Consul API: https://github.com/poppyred/python-consul2

[feoh]

I'm marking this "Needs triage" as I can't find a "Won't Fix" designation in Github issues :) Please close it you think it appropriate.

In the end analysis, once we got into the weeds with this, we realized that there are complications around secrets management that would make this entirely too cost prohibitive to consider at the moment.

Specifically:

• In order to do this right we'd need a single pipeline for each app with stages for CI, QA and Production, each talking to the appropriate Vault instance. Having one class of worker machine access all 3 Vault environments is problematic.
• We'd need to do the same for Consul

It may actually be easier to entirely get away from using Heroku than to build this.

[Ardiea]

In the end analysis, once we got into the weeds with this, we realized that there are complications around secrets management that would make this entirely too cost prohibitive to consider at the moment.

This is disappointing but the challenges you state make sense. 😞

[feoh]

Rather than moving Heroku settings management into Concourse, we should consider moving them into Pulumi with the Pulumi Heroku Provider.

[blarghmatey]

I'm closing this issue in favor of https://github.com/mitodl/ol-infrastructure/issues/2074