Closed datalass1 closed 5 years ago
fastai source to set up Google Cloud SDK fastai source to get back to work in GCP.
Command-line interface for Google Cloud Platform products and services. The Cloud SDK is a set of tools for Cloud Platform. It contains gcloud, gsutil, and bq, which you can use to access Google Compute Engine, Google Cloud Storage, Google BigQuery, and other products and services from the command-line. You can run these tools interactively or in your automated scripts.
gcloud manages authentication, local configuration, developer workflow, and interactions with the Cloud Platform APIs.
I already have an account. Skip this.
Install Google Cloud’s command line interface (CLI) software from Google.
# Create environment variable for correct distribution
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
# Add the Cloud SDK distribution URI as a package source
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
# Import the Google Cloud Platform public key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
# Update the package list and install the Cloud SDK
sudo apt-get update && sudo apt-get install google-cloud-sdk
source Google Cloud SDK documentation
Run gcloud init to start the authentication process. Hit enter when prompted.
gcloud init
If at this stage I create a new project rek-fastai Go into google cloud console Projects by billing account in manage billing to link billing account ID to the new project.
from error: ERROR: (gcloud.compute.instances.create) Could not fetch resource: - Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally
$ gcloud compute ssh --zone=$ZONE jupyter@$INSTANCE_NAME -- -L 8080:localhost:8080
In a web-browser go to: localhost:8080/tree Note that this only works while you maintain the ssh connection in your terminal.
Now your command line which should show a prompt along the lines of jupyter@my-fastai-instance
You should make sure Github is configured and pull from the repository. You can do this by typing the following lines:
cd tutorials/fastai/course-v3
git checkout .
git pull
You should also update the fastai library:
sudo /opt/anaconda3/bin/conda install -c fastai fastai
Preemptible instances: We are running a preemptible instance (notice the ‘–preemptible’ parameter in our command). A preemptible GCP instance is cheaper than traditional instances but it has two main disadvantages:
It can be preempted (stopped) with a 30 second notice at any time due to high demand. It will always be stopped after 24 hours of continuous running. If your instance is stopped, your saved data will be kept safe but if you are running a model, the progress will be lost.
Remove it and the price of the instance will go up. But better for deeper models if I move beyond beginner.
Providing that gcloud is installed, see above Step 2: Install Google CLI:
$ gcloud compute ssh --zone=$ZONE jupyter@$INSTANCE_NAME -- -L 8080:localhost:8080
So my fastai instance is:
$ gcloud compute ssh --zone="us-west2-b" jupyter@"my-fastai-instance" -- -L 8080:localhost:8080
Then connect to Jupyter notebook. In a web-browser go to: localhost:8080/tree
Hey Pesto! You are connected to a US server with awesome compute for fastai for FREE! (for now) :)
I have a Google Cloud Platform account. Previously I used the GeoCode API to change postcodes from a housing dataset to coordinates for mapping house prices in Somerset.
https://console.cloud.google.com/home/dashboard?project=geopackage-223116
I have £230.58 credit which will end on 9th November.