Deploy to Heroku with Google Drive secrets and GCP auth.json

[mfekadu]

Objective

Deploy to Heroku with Google Drive secrets and GCP auth.json because there's quite a few environment variables needed to make the system work.

• [x] auth.json for the nimbus-nlp
• [x] a special yaml file for the google drive
• [x] config.json for NimbusDatabase

Key Result

The Nimbus API system is deployed and running in the cloud.

Additional Details

We may also need

• [x] Code similar to this that can automatically generate the auth.json in the event that auth.json cannot be merged into config.json
• [x] To update the config_SAMPLE.json
• [x] To use PyYAML
• [x] to reference the secrets in the deployment yml of the .github/workflows folder

[mfekadu]

three files need to be generated at deploy time using GitHub Actions using the environment variables

yaml file looks like this....

# This file allows authentication with Google Drive for the nimbus API
# This enables uploading files to our Google Drive folder.
# DO NOT upload this file (or credentials.json) to github or any other public directory!
client_config_backend: settings
client_config:
  client_id: SECRET
  client_secret: SECRET

save_credentials: True
save_credentials_backend: file
save_credentials_file: credentials.json

get_refresh_token: True

oauth_scope:
  - https://www.googleapis.com/auth/drive.file
  - https://www.googleapis.com/auth/drive.install
  - https://www.googleapis.com/auth/drive
  - https://www.googleapis.com/auth/drive.metadata

needed for this

https://github.com/calpoly-csai/api/blob/4914b94c78134db27f427718f992a16832e8c9e0/flask_api.py#L10-L11

credentials.json looks like this... (but @snekiam suggested to just try putting the whole credentials string into the GitHub Secrets page && file dump on deploy time)

{
    "access_token": "SECRET",
    "client_id": "SECRET",
    "client_secret": "SECRET",
    "refresh_token": "SECRET",
    "token_expiry": "2020-01-10T01:24:09Z",
    "token_uri": "https://accounts.google.com/o/oauth2/token",
    "user_agent": null,
    "revoke_uri": "https://oauth2.googleapis.com/revoke",
    "id_token": null,
    "id_token_jwt": null,
    "token_response": {
        "access_token": "SECRET",
        "expires_in": 3600,
        "scope": "https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/drive.file https://www.googleapis.com/auth/drive.metadata https://www.googleapis.com/auth/drive.install",
        "token_type": "Bearer"
    },
    "scopes": ["https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/drive.metadata", "https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/drive.install"],
    "token_info_uri": "https://oauth2.googleapis.com/tokeninfo",
    "invalid": false,
    "_class": "OAuth2Credentials",
    "_module": "oauth2client.client"
}

there also exists a folder_id.txt file that is used by

https://github.com/calpoly-csai/api/blob/4914b94c78134db27f427718f992a16832e8c9e0/flask_api.py#L246

config.json

{
    "mysql": {
        "host": "HOSTNAME",
        "port": "PORT e.g. 3306",
        "user": "USERNAME",
        "password": "PASSWORD",
        "database": "DATABASE",
        "sql_dir": "sql",
        "create_file": "csai_nov_8_2019_create_script.min.sql"
    }
}

TODO

Fix these lines of code (remove && just let the workflow set the environment)

https://github.com/calpoly-csai/api/blob/030bb84460331f001f3adeb3f30502c74a5bf5ee/nimbus-nlp/NIMBUS_NLP.py#L52-L54

[mfekadu]

There is another file auth.json

{
  "type": "service_account",
  "project_id": "variable-extraction-266916",
  "private_key_id": "SECRET",
  "private_key": "SECRET",
  "client_email": "var-ext@variable-extraction-266916.iam.gserviceaccount.com",
  "client_id": "SECRET",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/var-ext%40variable-extraction-266916.iam.gserviceaccount.com"
}

there is also another thing for the NLP with Google Cloud NLP API

projects/SECRET/locations/us-central1/models/SECRET

[mfekadu]

relevant commits

• https://github.com/calpoly-csai/api/commit/7c239d5e27d8d5922b4711b1e975b558a0c34588

• https://github.com/calpoly-csai/api/commit/47a0f1f3ffcbac4f155f8f025625affb6504168a

• https://github.com/calpoly-csai/api/commit/dff98fb5f1e42cfca80cf35622e66ead0b0ae179

• https://github.com/calpoly-csai/api/commit/734779a10a3fe8b744d5faddb11ccae9ac8248a8

• https://github.com/calpoly-csai/api/commit/080bdb5fa821a379d4fc807affd62802b22ad7dd

• https://github.com/calpoly-csai/api/commit/5ec1d9bb05838baecf53c5f6e3400db3a796b346

• https://github.com/calpoly-csai/api/commit/0d99d7ef62bb899d22ce721af9531c2ed661d9ac

[mfekadu]

the deployed code is still buggy, but this issue is resolved because the deployment process itself works fine and the environment variables are set correctly and the needed files get generated automatically