search

FIRST-Tech-Challenge / fmltc

FIRST Machine Learning Toolchain
Other
38 stars 14 forks source link

readme

fmltc

FIRST Machine Learning Toolchain

Usage

Visit this file for a user guide.

Setup

Google Cloud

  1. Install the Google Cloud SDK

    See https://cloud.google.com/sdk/install for instructions.

  2. Create a Google Cloud Project.

  3. Enable billing

  4. Set the environment variable FMLTC_GCLOUD_PROJECT_ID

    FMLTC_GCLOUD_PROJECT_ID=<project id>
export FMLTC_GCLOUD_PROJECT_ID

  5. Set the Google Cloud Project ID

    gcloud config set project ${FMLTC_GCLOUD_PROJECT_ID}

  6. Enable APIs.

    gcloud services enable cloudfunctions.googleapis.com
gcloud services enable ml.googleapis.com
gcloud services enable secretmanager.googleapis.com
gcloud services enable compute.googleapis.com
gcloud services enable cloudbuild.googleapis.com

  7. Create a service account, generate a key.json file, and store it as a secret.\ Important! Make sure the current working directory is the fmltc directory when you run these commands.

    gcloud iam service-accounts create ${FMLTC_GCLOUD_PROJECT_ID}-service-account
gcloud projects add-iam-policy-binding ${FMLTC_GCLOUD_PROJECT_ID} --member "serviceAccount:${FMLTC_GCLOUD_PROJECT_ID}-service-account@${FMLTC_GCLOUD_PROJECT_ID}.iam.gserviceaccount.com" --role "roles/owner"
gcloud iam service-accounts keys create key.json --iam-account ${FMLTC_GCLOUD_PROJECT_ID}-service-account@${FMLTC_GCLOUD_PROJECT_ID}.iam.gserviceaccount.com
gcloud secrets create key_json --replication-policy="automatic" --data-file="key.json"

  8. Create a secret key for configuring flask. The secret key can be any text you want, but you should keep it private. Execute the following commands, replacing <YOUR-FLASK-SECRET-KEY> with the key you want to use.

    echo "<YOUR-FLASK-SECRET-KEY>" >flask_secret_key.txt
gcloud secrets create flask_secret_key --replication-policy="automatic" --data-file=flask_secret_key.txt
rm flask_secret_key.txt

  9. Give the App Engine default service account access to the secret.

    • [ ] Go to https://console.cloud.google.com/iam-admin/iam?project=my_project_id (replace my_project_id with your actual project ID)
    • [ ] Look for the row that shows App Engine default service account in the Name column.
    • [ ] At the far right of that row, click on the pencil icon (hint text says Edit principal)
    • [ ] Click + ADD ANOTHER ROLE
    • [ ] Under Select a role, where it says Type to filter enter secret accessor
    • [ ] Click Secret Manager Secret Accessor
    • [ ] Click SAVE

  10. Create cloud storage buckets.

    gsutil mb -c standard gs://${FMLTC_GCLOUD_PROJECT_ID}
gsutil defacl set public-read gs://${FMLTC_GCLOUD_PROJECT_ID}
gsutil mb -c standard gs://${FMLTC_GCLOUD_PROJECT_ID}-blobs
gsutil mb -c standard gs://${FMLTC_GCLOUD_PROJECT_ID}-action-parameters

  11. Create the Datastore.

  12. Grant the ml.serviceAgent role to your TPU service account.

    • [ ] Run the following command 
      curl -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
https://ml.googleapis.com/v1/projects/${FMLTC_GCLOUD_PROJECT_ID}:getConfig
    • [ ] Look for the tpuServiceAccount value in the curl command output.
    • [ ] Set the environment variable FMLTC_TPU_SERVICE_ACCOUNT 
      FMLTC_TPU_SERVICE_ACCOUNT=<tpu service account>
export FMLTC_TPU_SERVICE_ACCOUNT
    • [ ] Run the following command 
      gcloud projects add-iam-policy-binding ${FMLTC_GCLOUD_PROJECT_ID}  \
--member serviceAccount:${FMLTC_TPU_SERVICE_ACCOUNT} --role roles/ml.serviceAgent

Create and upload the team_info/teams file.

  1. Create a text file named teams containing one line for each team allowed to use the tools. Each line should look like this:

    <program>,<team number>,<team code>
    • Program must be FTC or FRC.
    • Team number should be the team number.
    • Team code should be the code that is given to that team. It can contain any characters and can be any length. It is in essence the user password.
    • It can be delimited by either , or ,

    Here's an example

    FTC, 25,    094e801d
FTC, 217,   676F6174
FTC, 4634,  65796573
FTC, 11115, 6629ab97
FRC, 67,    8bfef8bf
FRC, 254,   f929a006
FRC, 1678,  f67145cf
  2. Go to https://console.cloud.google.com/storage/browser/my_project_id-blobs?project=my_project_id (replace my_project_id with your actual project ID)
  3. Click Create folder. Enter team_info and click CREATE.
  4. Click team_info to go to https://console.cloud.google.com/storage/browser/my_project_id-blobs/team_info/?project=my_project_id (replace my_project_id with your actual project ID)
  5. Click Upload files. In the file chooser, select your teams file.

Clone the fmltc repo.

  1. In a browser, go to https://github.com/FIRST-Tech-Challenge/fmltc
  2. Click on the green Code button and figure out which way to clone the repo onto your local machine.
  3. After cloning the repo, cd fmltc

Install the Google Closure Compiler

Important! Make sure the current working directory is the fmltc directory when you run these commands.

mkdir -p ~/tmp_fmltc/
curl -o ~/tmp_fmltc/compiler-20200406.zip https://dl.google.com/closure-compiler/compiler-20200406.zip
mkdir ../closure-compiler
pushd ../closure-compiler
unzip ~/tmp_fmltc/compiler-20200406.zip
popd

Install the Google Closure Library

Important! Make sure the current working directory is the fmltc directory when you run these commands.

mkdir -p ~/tmp_fmltc/
curl -o ~/tmp_fmltc/closure-library_v20200406.zip https://codeload.github.com/google/closure-library/zip/refs/tags/v20200406
mkdir ../closure-library
pushd ../closure-library
unzip ~/tmp_fmltc/closure-library_v20200406.zip
popd

Install JDK

Depending on your OS and distribution there are various ways to install JDK. See https://www.oracle.com/java/technologies/downloads/ for instructions.

Docker

Install

Depending on your OS and distribution there are various ways to install Docker. https://docs.docker.com/get-docker/

Authenticate

gcloud auth configure-docker
Do you want to continue (Y/n)?  y

Fill in the values in server/app_engine/env_variables.yaml

  1. Replace <YOUR-PROJECT-ID> with the Google Cloud Project ID for your project.
  2. Replace <YOUR-ORIGIN> with the base URL that will serve the website.

Set Version

Important! Make sure the current working directory is the fmltc root directory when you run these commands.

The following command will set the version of fmltc displayed to the user to the current git commit hash.

echo "{ \"version\": \"$(git rev-parse --short HEAD)\" }" > server/app_engine/app.properties

If you wish the version to be set to something other than the current git commit hash you can use the following code replacing <VERSION_NAME> with the desired name of your version

echo "{ \"version\": \"<VERSION_NAME>\" }" > server/app_engine/app.properties

Setup the environment.

Important! Make sure the current working directory is the fmltc directory when you run these commands.

source env_setup.sh

Deploy everything.

Important! Make sure the current working directory is the fmltc directory when you run these commands.

  1. Deploy the Datastore indexes.

    source env_setup.sh
scripts/deploy_indexes.sh

  2. Deploy the static content.

    source env_setup.sh
scripts/deploy_static.sh

  3. Deploy the javascript code.

    source env_setup.sh
scripts/deploy_js.sh

  4. Deploy the Cloud Function.

    source env_setup.sh
scripts/deploy_cloud_function.sh
    • If you see the following, enter N. 
      Allow unauthenticated invocations of new function 
[perform_action]? (y/N)?

  5. Deploy the App Engine code.

    source env_setup.sh
scripts/deploy_gae.sh

  6. Deploy the object detection docker image.

    source env_setup.sh
scripts/deploy_docker_image.sh

Try it out

Go to the URL you found earlier at https://console.cloud.google.com/appengine?project=YOUR-PROJECT-ID (replace my_project_id with your actual project ID)