Closed snekiam closed 4 years ago
Note that before a deploy will work, some secrets need to be added: GCE_PROJECT, GIT_SSH_CERT, GOOGLE_APPLICATION_CREDENTIALS, PORT, and SA_EMAIL.
Also I apologize for the 80 commits, this took a while to get working...
I highly recommend smushing the 80 commits together,
git reset --soft HEAD~80 && git commit -m "blablabla"
should do the trick
Let's merge and see what happens!
And quick fixes to dev
if need be
also avoid the following code:
workers = multiprocessing.cpu_count() * 2 + 1
because I read somewhere that on virtual machines... that multiprocessing.cpu_count
doesn't really work
https://www.google.com/search?q=python+multiprocessing.cpu_count+virtual+mahcine
β api git:(pr/146) β ./gce_deploy.sh
invalid argument "gcr.io//-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
invalid reference format
./gce_deploy.sh: line 24: gcloud: command not found
https://cloud.google.com/sdk/docs/downloads-interactive
gmail account
and created a new project via command-line by choosing option 8
...
...
You are logged in as: [******@gmail.com].
Pick cloud project to use:
[1] ******-******
[2] ******-******-******
[3] ******-******-******
[4] ******-******-******
[5] quickstart-******
[6] ******-******-******
[7] ******-******-******
[8] Create a new project
Please enter numeric choice or text value (must exactly match list
item): 8
Enter a Project ID. Note that a Project ID CANNOT be changed later.
Project IDs must be 6-30 characters (lowercase ASCII, digits, or
hyphens) in length and start with a lowercase letter. mfekadu-test-pr-146
Waiting for [operations/cp.8279053178082129846] to finish...done.
Your current project has been set to: [mfekadu-test-pr-146].
...
...
β api git:(pr/146) β ./gce_deploy.sh
invalid argument "gcr.io//-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
invalid reference format
ERROR: (gcloud.compute.instances.update-container) argument --zone: expected one argument
Usage: gcloud compute instances update-container INSTANCE_NAME [optional flags]
optional flags may be --clear-container-args | --clear-container-command |
--container-arg | --container-command |
--container-env | --container-env-file |
--container-image | --container-mount-disk |
--container-mount-host-path | --container-mount-tmpfs |
--container-privileged | --container-restart-policy |
--container-stdin | --container-tty | --help |
--remove-container-env | --remove-container-mounts |
--zone
For detailed information on this command and its flags, run:
gcloud compute instances update-container --help
β api git:(pr/146) β cat gce_deploy.sh
#!/bin/bash
export GITHUB_SHA=`git rev-parse HEAD`
docker build -t gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA \
--build-arg GITHUB_SHA="$GITHUB_SHA" \
--build-arg GITHUB_REF="$GITHUB_REF" \
--build-arg DATABASE_HOSTNAME \
--build-arg DATABASE_PASSWORD \
--build-arg DATABASE_USERNAME \
--build-arg DATABASE_NAME \
--build-arg PYDRIVE_CLIENT_ID \
--build-arg PYDRIVE_CLIENT_SECRET \
--build-arg GOOGLE_DRIVE_CREDENTIALS \
--build-arg GOOGLE_DRIVE_FOLDER_ID \
--build-arg GOOGLE_CLOUD_NLP_CREDENTIALS \
--build-arg GOOGLE_CLOUD_NLP_MODEL_NAME \
--build-arg GIT_SSH_CERT \
--build-arg PORT .
docker push gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA
# sleep for 60 seconds to allow gce to restart after deploy
gcloud compute instances update-container $GCE_INSTANCE \
--zone $GCE_INSTANCE_ZONE \
--container-image=gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA \
--project=$GCE_PROJECT && sleep 60 && gcloud compute ssh $GCE_INSTANCE --zone=$GCE_INSTANCE_ZONE --project=$GCE_PROJECT --command='docker image prune -a -f'
β api git:(pr/146) β echo $GCE_INSTANCE_ZONE
β api git:(pr/146) β echo $GCE_INSTANCE
β api git:(pr/146) β cd ../nimbus-config-secrets
β nimbus-config-secrets git:(master) git pull
Updating 53a8410..b5cfce2
Fast-forward
.export_env_vars | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 57 insertions(+), 1 deletion(-)
β nimbus-config-secrets git:(master) source .export_env_vars
β nimbus-config-secrets git:(master) cd ../api
β api git:(pr/146) β ./gce_deploy.sh
Sending build context to Docker daemon 13.02MB
Step 1/43 : FROM ubuntu:latest
---> 72300a873c2c
Step 2/43 : ENV DEBIAN_FRONTEND=noninteractive
---> Running in d2940bdf6c86
Removing intermediate container d2940bdf6c86
---> f9fda1cdcb4b
Step 3/43 : RUN apt-get update && apt-get install -y python3-pip python3-dev certbot cron git && cd /usr/local/bin && ln -s /usr/bin/python3 python && pip3 install --upgrade pip && chmod 777 /usr/lib/python3/dist-packages/*
...
...
[Warning] One or more build-args [GITHUB_SHA GITHUB_REF] were not consumed
Successfully built dc4cdced5f97
Successfully tagged gcr.io/csai-nimbus/nimbus-api-test-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba
The push refers to repository [gcr.io/csai-nimbus/nimbus-api-test-image]
abb0508b91aa: Preparing
949f35d317c8: Preparing
2e5dc33b2170: Preparing
240d70be77b3: Preparing
18cf01e5f6bb: Preparing
16608376066d: Waiting
005a2d602086: Waiting
55e667623970: Waiting
9c6a583746eb: Waiting
1852b2300972: Waiting
03c9b9f537a4: Waiting
8c98131d2d1d: Waiting
cc4590d6a718: Waiting
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Updating specification of container [nimbus-api-test]...done.
Stopping instance [nimbus-api-test]...β Ή
...
...
Updating specification of container [nimbus-api-test]...done.
Stopping instance [nimbus-api-test]...done.
Starting instance [nimbus-api-test]...done.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
...
...
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/mfekadu/.ssh/google_compute_engine.
Your public key has been saved in /Users/mfekadu/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:P94lGSi7zsfQp+xum4u92gmZP0xT43ZnKMP6K/3+Eqo mfekadu@mf.local
The key's randomart image is:
+---[RSA 3072]----+
| |
| |
| |
| .o |
| S..+.. . |
| .*+ Bo+ o|
| =*oBo=.+ |
| . O@=oo. |
| .**EO++oo.|
+----[SHA256]-----+
Updating project ssh metadata...β ΆUpdated [https://www.googleapis.com/compute/v1/projects/csai-nimbus].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.4404804429545858063' (ED25519) to the list of known hosts.
Enter passphrase for key '/Users/mfekadu/.ssh/google_compute_engine':
Enter passphrase for key '/Users/mfekadu/.ssh/google_compute_engine':
Deleted Images:
untagged: gcr.io/gce-containers/konlet:v.0.9-latest
untagged: gcr.io/gce-containers/konlet@sha256:7070409b900039bb1b33bb76ba0c8d8198e456b8da3c3a4992cd51b56f3fdf7e
deleted: sha256:da64965a2b28bf1169e21791c29e7a1211c57e90a2d7a2b38ee43d8445943b53
deleted: sha256:9a8ce936fa823d31177b9276befc59c509bf238ba3382a6aae3406dd05bf4b66
deleted: sha256:836b7ef9870ee5878e596727acd7bae91a14aa3ee5fcc46b4a2ff7a63d37fb2c
deleted: sha256:1fe8fbac6f7b1118d06ab211858f77314e117536d4b8928fe69a2b12b621f840
untagged: gcr.io/csai-nimbus/nimbus-api-test-image:be01d52acb429b516f2a0773d335db46b13b31d3
untagged: gcr.io/csai-nimbus/nimbus-api-test-image@sha256:4507ab9daaa02d7c6c82bffa038679525f15c3b655674d9d7656271a9357d971
deleted: sha256:61c7d44d474cb306368224278ae875e967ff6e2f80742cfc5715c32b02b6902b
deleted: sha256:e0f2d76b0f97f26e67b8e892ea2884777170c716a2cd2642ef7fb870e10ac641
deleted: sha256:ef1c002d83787f7f46512298f20a2087dbb13a01b0ce06598544c4422a6f652c
deleted: sha256:3073afdbf8f5bda76a9ecac4ecdffd8c94866f0aee5152dbd82e186964a970d3
deleted: sha256:700f5fa961326a1e6b00ddb248bbc520cafe1c85b8ddac78a62c5c279b83bd31
deleted: sha256:be4c7f74ce1d40af2fede787126441ef31a5e935bed5e969f3337acda9e40ca0
deleted: sha256:e6996084462c8820df0a8038023483c6e7488f13c5ee303370da8dbfe62b96e7
deleted: sha256:cab954af199cfa807e9c65c22308bf8ea655e9aefa3abf4191d28c4a76aa2584
deleted: sha256:3f1b47aad30968e17896c25a9e9d5e881fe861529aa9f649eeb4a5928c3e5571
deleted: sha256:d04385d6248bc701c2aaef0711bfce5a44050911b7690214e51d1f67dc06ba4a
Total reclaimed space: 1.067GB
β api git:(pr/146) β
mfekadu-test-pr-146
images deleted??
That's from the gcloud compute ssh $GCE_INSTANCE --zone=$GCE_INSTANCE_ZONE --project=$GCE_PROJECT --command='docker image prune -a -f'
step. Docker will prune any images which are not in use.
@snekiam let's merge because we now have a
and since it was super easy for me to go through the steps, I believe the automatic GitHub Action deployment will go smoothly.
I think I have to update the ssh cert secret first but then I'll merge it.
Success!
Very cool!
What's New?
There's a bunch of commits, but the changes are pretty much:
The following are the code changes which are unrelated to google cloud, and should (probably) be in a separate PR. If it bothers people, I can split it out, but they're pretty small changes.
The following changes will help us migrate to Python 3.8, as well as a newer version of sklearn, spacy, etc. This is probably a good idea to do at some point, so I figure we can do it now since this is a big change, and it lets us use ubuntu:latest as our docker image.
Some things I'm not sure about
We'll have to update the API endpoints to point to
https://nimbus.api.calpolycsai.com
, since this will stop deploying to heroku.Fixes #133, #15
Type of change (pick-one)
How Has This Been Tested?
Ran on python 3.7.5 locally, queried with questions using postman Ran on python 3.8 on google cloud (using github actions to deploy), queried with questions using postman Ran ./run_tests.sh
I did notice that after ~8 hours, our database connections failed. Unfortunately, heroku isn't working right now so I can't tell if this is the same issue that was there from logs. I put this in #145, since I don't think its an issue related to this change.
Checklist (check-all-before-merge)
formatting help:
- [x]
means "checked' and- [ ]
means "unchecked"[ ] I documented my code according to the Google Python Style Guide
[ ] I ran
./build_docs.sh
and the docs look fine[ ] I ran
./type_check.sh
and got no errors[ ] I ran
./format.sh
because it automatically cleans my code for me π[ ] I ran
./lint.sh
to check for what "format" missed[ ] I added my tests to the
/tests
directory[x] I ran
./run_tests.sh
and all the tests pass