search

mariokostelac / sagemaker-knockout

🥊 Automatic shutdown for SageMaker instance when it's not active!
MIT License
9 stars 1 forks source link

Send logs to cloudwatch #4

Open mariokostelac opened 5 years ago

mariokostelac commented 5 years ago

We currently store logs in ~/SageMaker/sagemaker_knockout.log. It would be nice to send them to CloudWatch as well.

mariokostelac commented 5 years ago

How to do that?

Add following snippet to /etc/awslogs/awslogs.conf:

[/home/ec2-user/SageMaker/sagemaker_knockout.log]
file = /home/ec2-user/SageMaker/sagemaker_knockout.log*
buffer_duration = 5000
log_stream_name = mario-learning/SageMakerKnockout
initial_position = end_of_file
log_group_name = /aws/sagemaker/NotebookInstances

Replace mario-learning with instance name.

Running just sudo service awslogs restart did not work, but killing the logs process and then running sudo service awslogs restart did work.

mariokostelac commented 5 years ago

Turns out that the restart does not work because /etc/init.d/awslogs has a bug in how it checks for the script. It checks for /usr/bin/python2.6 $prog, but should be really checking it with /usr/bin/python3.6 $prog. Have to figure out how to report that to AWS.