aws-samples / amazon-sagemaker-notebook-instance-lifecycle-config-samples

A collection of sample scripts to customize Amazon SageMaker Notebook Instances using Lifecycle Configurations
MIT No Attribution
416 stars 248 forks source link

#90 check notebook and terminal sessions #114

Open takeru1205 opened 9 months ago

takeru1205 commented 9 months ago

Issue #, if available: Issue: https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/issues/90

Description of changes: The current sample lifecycle configuration only checks for notebook sessions. I have updated it to also monitor terminal sessions. This enhancement is vital as the instance will not stop if there are any non-idle notebook sessions.

Testing Done

Idle time is set to 10 minutes.

*/5 * * * * /home/ec2-user/anaconda3/bin/python3 /tmp/autostop.py --time 600 --ignore-connections>> /var/log/jupyter.log

Logs showing kernel activity:

  | 2023-12-12T11:52:04.602+09:00 | [I 2023-12-12 02:51:59.963 ServerApp] New terminal with automatic name: 1
  | 2023-12-12T11:52:20.603+09:00 | [I 2023-12-12 02:52:15.685 ServerApp] New terminal with automatic name: 2
  | 2023-12-12T11:55:02.391+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:51:43.955000
  | 2023-12-12T11:55:02.391+09:00 | Notebook idle state set as False since no sessions detected.
  | 2023-12-12T11:55:02.391+09:00 | {'name': '1', 'last_activity': '2023-12-12T02:53:44.396492Z'}
  | 2023-12-12T11:55:02.391+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:53:44.396492
  | 2023-12-12T11:55:02.391+09:00 | {'name': '2', 'last_activity': '2023-12-12T02:53:00.323036Z'}
  | 2023-12-12T11:55:02.391+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:53:00.323036
  | 2023-12-12T11:55:06.602+09:00 | Kernel not idle. Pass.
  | 2023-12-12T12:00:01.980+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:51:43.955000
  | 2023-12-12T12:00:01.980+09:00 | Notebook idle state set as False since no sessions detected.
  | 2023-12-12T12:00:01.980+09:00 | {'name': '1', 'last_activity': '2023-12-12T02:53:44.396492Z'}
  | 2023-12-12T12:00:01.980+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:53:44.396492
  | 2023-12-12T12:00:01.980+09:00 | {'name': '2', 'last_activity': '2023-12-12T02:55:38.896191Z'}
  | 2023-12-12T12:00:01.980+09:00 | Kernel is not idle. Last activity time = 2023-12-12 02:55:38.896191
  | 2023-12-12T12:00:06.602+09:00 | Kernel not idle. Pass.
  | 2023-12-12T12:05:02.021+09:00 | Kernel is idle. Last activity time = 2023-12-12 02:51:43.955000

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

dmlause commented 6 months ago

Thank you for raising this PR! Could you test this again please since it has been some time since this was originally raised?