search

googledatalab / datalab

Interactive tools and developer experiences for Big Data on Google Cloud Platform.
Apache License 2.0
975 stars 250 forks source link

How to use Monitoring Agent for Datalab Instance #1925

Open majidshokrolahi opened 6 years ago

majidshokrolahi commented 6 years ago

I need to monitor all our Datalab instances. I know by using Monitoirng Agent we can monitor VMs with many metrics and i need to install it on the Datalab instances. Is it possibile to install it on the Datalab vm as it is a Chromeos vm ? if yes, how ?

chmeyers commented 6 years ago

The monitoring agent does not currently support the Container-Optimized OS that is used by Datalab. https://cloud.google.com/monitoring/agent/#supported_operating_systems

Even without the agent, Stackdriver will still pull CPU, Disk, and Network usage from the VMs, you just don't get the other metrics , most of which aren't applicable to Datalab. You'll miss out on RAM, swapfile usage, and individual process metrics.

I think the only way around this currently would be to run the Datalab Docker container on a non-ContainerOS image, which would mean writing your own VM startup script.

chmeyers commented 6 years ago

Looking at this though, I wonder why we don't configure docker to send it's logs to gcp: https://cloud.google.com/community/tutorials/docker-gcplogs-driver#configuring-the-google-cloud-logging-driver-for-a-single-container

majidshokrolahi commented 6 years ago

@chmeyers : In fact in our project we need these two metrics: RAM, swapfile usage. About the workaround you have mentioned, would you give me more info ? for example where I can find the last docker image of Datalab ? any help is appriciated.

chmeyers commented 6 years ago

The latest "released" Docker image is always gcr.io/cloud-datalab/datalab:latest There are also daily builds with tags that look like gcr.io/cloud-datalab/datalab:local-20180124, but I'd recommend using 'latest'

Prior to #1888, the GPU VMs used an Ubuntu host image, so you could basically take that script: https://github.com/googledatalab/datalab/blob/2625483e183b5a57ced93ffcdf6c869fd59760d3/tools/cli/commands/creategpu.py, remove out the GPU related stuff, and add the monitoring agent to the _DATALAB_STARTUP_SCRIPT. (Or keep the GPU stuff if you want to use GPUs)