snekiam commented 4 years ago

What's New?

There's a bunch of commits, but the changes are pretty much:

Added a .dockerignore file (interesting things are at the very bottom of it, the rest was a template I believe)
Temporarily disabled heroku staging deploy (didn't want it trying deploying from my fork, after all)
Added gce.yaml to deploy to Google cloud, as well as build the docker image, etc for google cloud
Modified the dockerfile to use ubuntu latest, as well as several other changes for google cloud (this will probably break heroku deploys, another reason I disabled that workflow)
Added a script to pull letsencrypt configs down on deploy
Added scripts for keeping letsencrypt updated (cronjob, as well as on push)

The following are the code changes which are unrelated to google cloud, and should (probably) be in a separate PR. If it bothers people, I can split it out, but they're pretty small changes.

Removed some duplicate code in database_wrapper.py
Changed nimbus.py's init to train the model if it doesn't exist - this is probably better than crashing, which is what we currently do
Fixed a bug with train_model from QuestionClassifier, and removed some duplicate code

The following changes will help us migrate to Python 3.8, as well as a newer version of sklearn, spacy, etc. This is probably a good idea to do at some point, so I figure we can do it now since this is a big change, and it lets us use ubuntu:latest as our docker image.

joblib is no longer in sklearn.externals
numpy, scikit-learn, spacy, srsly, and thinc are all slightly newer versions. As a result, models trained on older versions are no longer compatible.

Some things I'm not sure about

overall_features.json is different. I'm not sure why, but this is generated when you save the model. Some basic questions seem to work against the API, but maybe someone can speak to this.

We'll have to update the API endpoints to point to https://nimbus.api.calpolycsai.com, since this will stop deploying to heroku.

Fixes #133, #15

Type of change (pick-one)

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?

Ran on python 3.7.5 locally, queried with questions using postman Ran on python 3.8 on google cloud (using github actions to deploy), queried with questions using postman Ran ./run_tests.sh

I did notice that after ~8 hours, our database connections failed. Unfortunately, heroku isn't working right now so I can't tell if this is the same issue that was there from logs. I put this in #145, since I don't think its an issue related to this change.

Checklist (check-all-before-merge)

formatting help: - [x] means "checked' and - [ ] means "unchecked"

[ ] I documented my code according to the Google Python Style Guide
[ ] I ran ./build_docs.sh and the docs look fine
[ ] I ran ./type_check.sh and got no errors
[ ] I ran ./format.sh because it automatically cleans my code for me 😄
[ ] I ran ./lint.sh to check for what "format" missed
[ ] I added my tests to the /tests directory
[x] I ran ./run_tests.sh and all the tests pass

snekiam commented 4 years ago

Note that before a deploy will work, some secrets need to be added: GCE_PROJECT, GIT_SSH_CERT, GOOGLE_APPLICATION_CREDENTIALS, PORT, and SA_EMAIL.

snekiam commented 4 years ago

Also I apologize for the 80 commits, this took a while to get working...

Jason-Ku commented 4 years ago

I highly recommend smushing the 80 commits together,

git reset --soft HEAD~80 && git commit -m "blablabla"

should do the trick

mfekadu commented 4 years ago

Let's merge and see what happens!

And quick fixes to dev if need be

mfekadu commented 4 years ago

also avoid the following code:

workers = multiprocessing.cpu_count() * 2 + 1

because I read somewhere that on virtual machines... that multiprocessing.cpu_count doesn't really work

https://www.google.com/search?q=python+multiprocessing.cpu_count+virtual+mahcine

mfekadu commented 4 years ago

this comment documents my review process...

Troubleshooting...

➜  api git:(pr/146) ✗ ./gce_deploy.sh
invalid argument "gcr.io//-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
invalid reference format
./gce_deploy.sh: line 24: gcloud: command not found

following instructions here...

https://cloud.google.com/sdk/docs/downloads-interactive

Interactive installation

wow that was easy

logged in with a `gmail account` and created a new project via command-line by choosing option `8`

...
...

You are logged in as: [******@gmail.com].

Pick cloud project to use:
 [1] ******-******
 [2] ******-******-******
 [3] ******-******-******
 [4] ******-******-******
 [5] quickstart-******
 [6] ******-******-******
 [7] ******-******-******
 [8] Create a new project
Please enter numeric choice or text value (must exactly match list
item):  8

Enter a Project ID. Note that a Project ID CANNOT be changed later.
Project IDs must be 6-30 characters (lowercase ASCII, digits, or
hyphens) in length and start with a lowercase letter. mfekadu-test-pr-146
Waiting for [operations/cp.8279053178082129846] to finish...done.
Your current project has been set to: [mfekadu-test-pr-146].
...
...

try again

➜  api git:(pr/146) ✗ ./gce_deploy.sh
invalid argument "gcr.io//-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
invalid reference format
ERROR: (gcloud.compute.instances.update-container) argument --zone: expected one argument
Usage: gcloud compute instances update-container INSTANCE_NAME [optional flags]
  optional flags may be  --clear-container-args | --clear-container-command |
                         --container-arg | --container-command |
                         --container-env | --container-env-file |
                         --container-image | --container-mount-disk |
                         --container-mount-host-path | --container-mount-tmpfs |
                         --container-privileged | --container-restart-policy |
                         --container-stdin | --container-tty | --help |
                         --remove-container-env | --remove-container-mounts |
                         --zone

For detailed information on this command and its flags, run:
  gcloud compute instances update-container --help

realizing that I am missing some secrets

➜  api git:(pr/146) ✗ cat gce_deploy.sh
#!/bin/bash

export GITHUB_SHA=`git rev-parse HEAD`

docker build -t gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA \
          --build-arg GITHUB_SHA="$GITHUB_SHA" \
          --build-arg GITHUB_REF="$GITHUB_REF" \
          --build-arg DATABASE_HOSTNAME \
          --build-arg DATABASE_PASSWORD \
          --build-arg DATABASE_USERNAME \
          --build-arg DATABASE_NAME \
          --build-arg PYDRIVE_CLIENT_ID \
          --build-arg PYDRIVE_CLIENT_SECRET \
          --build-arg GOOGLE_DRIVE_CREDENTIALS \
          --build-arg GOOGLE_DRIVE_FOLDER_ID \
          --build-arg GOOGLE_CLOUD_NLP_CREDENTIALS \
          --build-arg GOOGLE_CLOUD_NLP_MODEL_NAME \
          --build-arg GIT_SSH_CERT \
          --build-arg PORT .

docker push gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA

# sleep for 60 seconds to allow gce to restart after deploy
gcloud compute instances update-container $GCE_INSTANCE \
    --zone $GCE_INSTANCE_ZONE \
    --container-image=gcr.io/$GCE_PROJECT/$GCE_INSTANCE-image:$GITHUB_SHA \
    --project=$GCE_PROJECT && sleep 60 && gcloud compute ssh $GCE_INSTANCE --zone=$GCE_INSTANCE_ZONE --project=$GCE_PROJECT --command='docker image prune -a -f'
➜  api git:(pr/146) ✗ echo $GCE_INSTANCE_ZONE

➜  api git:(pr/146) ✗ echo $GCE_INSTANCE

accessing nimbus-config-secrets

➜  api git:(pr/146) ✗ cd ../nimbus-config-secrets
➜  nimbus-config-secrets git:(master) git pull
Updating 53a8410..b5cfce2
Fast-forward
 .export_env_vars | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)
➜  nimbus-config-secrets git:(master) source .export_env_vars
➜  nimbus-config-secrets git:(master) cd ../api

it's working

➜  api git:(pr/146) ✗ ./gce_deploy.sh
Sending build context to Docker daemon  13.02MB
Step 1/43 : FROM ubuntu:latest
 ---> 72300a873c2c
Step 2/43 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Running in d2940bdf6c86
Removing intermediate container d2940bdf6c86
 ---> f9fda1cdcb4b
Step 3/43 : RUN apt-get update   && apt-get install -y python3-pip python3-dev certbot cron git   && cd /usr/local/bin   && ln -s /usr/bin/python3 python   && pip3 install --upgrade pip   && chmod 777 /usr/lib/python3/dist-packages/*
...
...

uh oh... permission issues

[Warning] One or more build-args [GITHUB_SHA GITHUB_REF] were not consumed
Successfully built dc4cdced5f97
Successfully tagged gcr.io/csai-nimbus/nimbus-api-test-image:ac5802aaa90d3d156b2f19db183714ccecf9dfba
The push refers to repository [gcr.io/csai-nimbus/nimbus-api-test-image]
abb0508b91aa: Preparing
949f35d317c8: Preparing
2e5dc33b2170: Preparing
240d70be77b3: Preparing
18cf01e5f6bb: Preparing
16608376066d: Waiting
005a2d602086: Waiting
55e667623970: Waiting
9c6a583746eb: Waiting
1852b2300972: Waiting
03c9b9f537a4: Waiting
8c98131d2d1d: Waiting
cc4590d6a718: Waiting
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Updating specification of container [nimbus-api-test]...done.
Stopping instance [nimbus-api-test]...⠹

asking me for a passphrase

...
...
Updating specification of container [nimbus-api-test]...done.
Stopping instance [nimbus-api-test]...done.
Starting instance [nimbus-api-test]...done.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
...
...

continuing with a passphrase... need another one

WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Your identification has been saved in /Users/mfekadu/.ssh/google_compute_engine.
Your public key has been saved in /Users/mfekadu/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:P94lGSi7zsfQp+xum4u92gmZP0xT43ZnKMP6K/3+Eqo mfekadu@mf.local
The key's randomart image is:
+---[RSA 3072]----+
|                 |
|                 |
|                 |
|           .o    |
|        S..+.. . |
|        .*+ Bo+ o|
|        =*oBo=.+ |
|       . O@=oo.  |
|       .**EO++oo.|
+----[SHA256]-----+
Updating project ssh metadata...⠶Updated [https://www.googleapis.com/compute/v1/projects/csai-nimbus].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.4404804429545858063' (ED25519) to the list of known hosts.
Enter passphrase for key '/Users/mfekadu/.ssh/google_compute_engine':

images deleted??

Enter passphrase for key '/Users/mfekadu/.ssh/google_compute_engine':
Deleted Images:
untagged: gcr.io/gce-containers/konlet:v.0.9-latest
untagged: gcr.io/gce-containers/konlet@sha256:7070409b900039bb1b33bb76ba0c8d8198e456b8da3c3a4992cd51b56f3fdf7e
deleted: sha256:da64965a2b28bf1169e21791c29e7a1211c57e90a2d7a2b38ee43d8445943b53
deleted: sha256:9a8ce936fa823d31177b9276befc59c509bf238ba3382a6aae3406dd05bf4b66
deleted: sha256:836b7ef9870ee5878e596727acd7bae91a14aa3ee5fcc46b4a2ff7a63d37fb2c
deleted: sha256:1fe8fbac6f7b1118d06ab211858f77314e117536d4b8928fe69a2b12b621f840
untagged: gcr.io/csai-nimbus/nimbus-api-test-image:be01d52acb429b516f2a0773d335db46b13b31d3
untagged: gcr.io/csai-nimbus/nimbus-api-test-image@sha256:4507ab9daaa02d7c6c82bffa038679525f15c3b655674d9d7656271a9357d971
deleted: sha256:61c7d44d474cb306368224278ae875e967ff6e2f80742cfc5715c32b02b6902b
deleted: sha256:e0f2d76b0f97f26e67b8e892ea2884777170c716a2cd2642ef7fb870e10ac641
deleted: sha256:ef1c002d83787f7f46512298f20a2087dbb13a01b0ce06598544c4422a6f652c
deleted: sha256:3073afdbf8f5bda76a9ecac4ecdffd8c94866f0aee5152dbd82e186964a970d3
deleted: sha256:700f5fa961326a1e6b00ddb248bbc520cafe1c85b8ddac78a62c5c279b83bd31
deleted: sha256:be4c7f74ce1d40af2fede787126441ef31a5e935bed5e969f3337acda9e40ca0
deleted: sha256:e6996084462c8820df0a8038023483c6e7488f13c5ee303370da8dbfe62b96e7
deleted: sha256:cab954af199cfa807e9c65c22308bf8ea655e9aefa3abf4191d28c4a76aa2584
deleted: sha256:3f1b47aad30968e17896c25a9e9d5e881fe861529aa9f649eeb4a5928c3e5571
deleted: sha256:d04385d6248bc701c2aaef0711bfce5a44050911b7690214e51d1f67dc06ba4a

Total reclaimed space: 1.067GB
➜  api git:(pr/146) ✗

not authorized to click here

enabled billing on Compute Engine VM Instances in gcloud web interface for the project `mfekadu-test-pr-146`

no VM there... hmm...

but it is running on a separate project

snekiam commented 4 years ago

images deleted??

That's from the gcloud compute ssh $GCE_INSTANCE --zone=$GCE_INSTANCE_ZONE --project=$GCE_PROJECT --command='docker image prune -a -f' step. Docker will prune any images which are not in use.

mfekadu commented 4 years ago

@snekiam let's merge because we now have a

a good script for the deployment to GCE
some documentation on the deployment to GCE

and since it was super easy for me to go through the steps, I believe the automatic GitHub Action deployment will go smoothly.

snekiam commented 4 years ago

I think I have to update the ssh cert secret first but then I'll merge it.

snekiam commented 4 years ago

Success!

mfekadu commented 4 years ago

Very cool!

calpoly-csai / api

Deploy to google cloud #146

What's New?

Fixes #133, #15

Type of change (pick-one)

How Has This Been Tested?

Checklist (check-all-before-merge)

this comment documents my review process...

Troubleshooting...

following instructions here...

wow that was easy

logged in with a `gmail account` and created a new project via command-line by choosing option `8`

try again

realizing that I am missing some secrets

accessing nimbus-config-secrets

it's working

uh oh... permission issues

asking me for a passphrase

continuing with a passphrase... need another one

images deleted??

not authorized to click here

enabled billing on Compute Engine VM Instances in gcloud web interface for the project `mfekadu-test-pr-146`

no VM there... hmm...

but it is running on a separate project

images deleted??

calpoly-csai / api

Deploy to google cloud #146

What's New?

Fixes #133, #15

Type of change (pick-one)

How Has This Been Tested?

Checklist (check-all-before-merge)

this comment documents my review process...

Troubleshooting...

following instructions here...

wow that was easy

logged in with a gmail account and created a new project via command-line by choosing option 8

try again

realizing that I am missing some secrets

accessing nimbus-config-secrets

it's working

uh oh... permission issues

asking me for a passphrase

continuing with a passphrase... need another one

images deleted??

not authorized to click here

enabled billing on Compute Engine VM Instances in gcloud web interface for the project mfekadu-test-pr-146

no VM there... hmm...

but it is running on a separate project

images deleted??

logged in with a `gmail account` and created a new project via command-line by choosing option `8`

enabled billing on Compute Engine VM Instances in gcloud web interface for the project `mfekadu-test-pr-146`