search

FIRST-Tech-Challenge / fmltc

FIRST Machine Learning Toolchain
Other
39 stars 14 forks source link

Use the google cloud secret manager to store key.json. #116

Closed lizlooney closed 3 years ago

cmacfarl commented 3 years ago

The key is still going to end up in the filesystem on the flask server until, or unless, we move it on the actions runner that does the app engine deploy out of the server directory.

Also, we need to do the same for client_secrets.json which is the authentication for the oidc integration with ftc-scoring.

lizlooney commented 3 years ago

The key is still going to end up in the filesystem on the flask server until, or unless, we move it on the actions runner that does the app engine deploy out of the server directory.

Why? I don't understand. It is never in the server directory. Right?

cmacfarl commented 3 years ago

Why? I don't understand. It is never in the server directory. Right?

It's the Install Key step in the terraform.yml. That drops the key in the server directory on the runner and the terraform apply, on merge, packages up that directory for a push to the app server source storage bucket. That's what those two google_storage_bucket_object resource entries in the terraform plan are showing. The files that get zipped there are the files on the runner.

If you don't mind me pushing straight to this branch, I can make this change tomorrow.

lizlooney commented 3 years ago

I don't mind you pushing to this branch.

github-actions[bot] commented 3 years ago

Terraform plan Succeeded for Workspace: default

Show Output ```diff An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create ! update in-place - destroy -/+ destroy and then create replacement Terraform will perform the following actions: # module.dev.google_app_engine_standard_app_version.fmltc-app-v1 will be updated in-place ! resource "google_app_engine_standard_app_version" "fmltc-app-v1" { id = "apps/ftc-ml-firstinspires-dev/services/default/versions/v1" name = "apps/ftc-ml-firstinspires-dev/services/default/versions/v1" # (9 unchanged attributes hidden) ! deployment { ! zip { ! source_url = "https://storage.googleapis.com/ftc-ml-firstinspires-dev-gae-source/83cea86df869eccf192c97c95b291893.zip" -> "https://storage.googleapis.com/ftc-ml-firstinspires-dev-gae-source/f56b39c8459f1db7165bb7fa00231920.zip" # (1 unchanged attribute hidden) } } - handlers { - auth_fail_action = "AUTH_FAIL_ACTION_REDIRECT" -> null - login = "LOGIN_OPTIONAL" -> null - security_level = "SECURE_OPTIONAL" -> null - url_regex = ".*" -> null - script { - script_path = "auto" -> null } } # (8 unchanged blocks hidden) } # module.dev.google_cloudfunctions_function.frame-extraction will be destroyed - resource "google_cloudfunctions_function" "frame-extraction" { - available_memory_mb = 8192 -> null - description = "Extracts frames after a video upload" -> null - entry_point = "perform_action" -> null - environment_variables = { - "project_id" = "ftc-ml-firstinspires-dev" } -> null - id = "projects/ftc-ml-firstinspires-dev/locations/us-central1/functions/perform_action" -> null - ingress_settings = "ALLOW_ALL" -> null - labels = {} -> null - max_instances = 0 -> null - name = "perform_action" -> null - project = "ftc-ml-firstinspires-dev" -> null - region = "us-central1" -> null - runtime = "python39" -> null - service_account_email = "ftc-ml-firstinspires-dev@appspot.gserviceaccount.com" -> null - source_archive_bucket = "ftc-ml-firstinspires-dev-gcf-source" -> null - source_archive_object = "505cffa8aae34ab1eb5a4adcc1687501.zip" -> null - timeout = 540 -> null - event_trigger { - event_type = "google.storage.object.finalize" -> null - resource = "projects/ftc-ml-firstinspires-dev/buckets/ftc-ml-firstinspires-dev-action-parameters" -> null - failure_policy { - retry = false -> null } } - timeouts { - create = "60m" -> null - update = "60m" -> null } } # module.dev.google_cloudfunctions_function.perform-action will be created + resource "google_cloudfunctions_function" "perform-action" { + available_memory_mb = 8192 + description = "Performs long running actions, such as extracting frames after a video upload" + entry_point = "perform_action" + environment_variables = { + "project_id" = "ftc-ml-firstinspires-dev" } + https_trigger_url = (known after apply) + id = (known after apply) + ingress_settings = "ALLOW_ALL" + max_instances = 0 + name = "perform_action" + project = (known after apply) + region = (known after apply) + runtime = "python39" + service_account_email = (known after apply) + source_archive_bucket = "ftc-ml-firstinspires-dev-gcf-source" + source_archive_object = "cc9b8e2ab7acc38b2f59c300d13a8c7d.zip" + timeout = 540 + vpc_connector_egress_settings = (known after apply) + event_trigger { + event_type = "google.storage.object.finalize" + resource = "ftc-ml-firstinspires-dev-action-parameters" + failure_policy { + retry = (known after apply) } } + timeouts { + create = "60m" + update = "60m" } } # module.dev.google_storage_bucket_object.app-server-archive must be replaced -/+ resource "google_storage_bucket_object" "app-server-archive" { ! content_type = "application/zip" -> (known after apply) ! crc32c = "Nm69tQ==" -> (known after apply) ! detect_md5hash = "g86obfhp7M8ZLJfJWykYkw==" -> "different hash" # forces replacement - event_based_hold = false -> null ! id = "ftc-ml-firstinspires-dev-gae-source-83cea86df869eccf192c97c95b291893.zip" -> (known after apply) + kms_key_name = (known after apply) ! md5hash = "g86obfhp7M8ZLJfJWykYkw==" -> (known after apply) ! media_link = "https://storage.googleapis.com/download/storage/v1/b/ftc-ml-firstinspires-dev-gae-source/o/83cea86df869eccf192c97c95b291893.zip?generation=1633153140722693&alt=media" -> (known after apply) - metadata = {} -> null ! name = "83cea86df869eccf192c97c95b291893.zip" -> "f56b39c8459f1db7165bb7fa00231920.zip" # forces replacement ! output_name = "83cea86df869eccf192c97c95b291893.zip" -> (known after apply) ! self_link = "https://www.googleapis.com/storage/v1/b/ftc-ml-firstinspires-dev-gae-source/o/83cea86df869eccf192c97c95b291893.zip" -> (known after apply) ! storage_class = "STANDARD" -> (known after apply) - temporary_hold = false -> null # (2 unchanged attributes hidden) } # module.dev.google_storage_bucket_object.cloud-function-archive must be replaced -/+ resource "google_storage_bucket_object" "cloud-function-archive" { ! content_type = "application/zip" -> (known after apply) ! crc32c = "6SL2nQ==" -> (known after apply) ! detect_md5hash = "UFz/qKrjSrHrWkrcwWh1AQ==" -> "different hash" # forces replacement - event_based_hold = false -> null ! id = "ftc-ml-firstinspires-dev-gcf-source-505cffa8aae34ab1eb5a4adcc1687501.zip" -> (known after apply) + kms_key_name = (known after apply) ! md5hash = "UFz/qKrjSrHrWkrcwWh1AQ==" -> (known after apply) ! media_link = "https://storage.googleapis.com/download/storage/v1/b/ftc-ml-firstinspires-dev-gcf-source/o/505cffa8aae34ab1eb5a4adcc1687501.zip?generation=1633153140690472&alt=media" -> (known after apply) - metadata = {} -> null ! name = "505cffa8aae34ab1eb5a4adcc1687501.zip" -> "cc9b8e2ab7acc38b2f59c300d13a8c7d.zip" # forces replacement ! output_name = "505cffa8aae34ab1eb5a4adcc1687501.zip" -> (known after apply) ! self_link = "https://www.googleapis.com/storage/v1/b/ftc-ml-firstinspires-dev-gcf-source/o/505cffa8aae34ab1eb5a4adcc1687501.zip" -> (known after apply) ! storage_class = "STANDARD" -> (known after apply) - temporary_hold = false -> null # (2 unchanged attributes hidden) } Plan: 3 to add, 1 to change, 3 to destroy. ```