aws-samples / amazon-sagemaker-notebook-instance-lifecycle-config-samples

A collection of sample scripts to customize Amazon SageMaker Notebook Instances using Lifecycle Configurations
MIT No Attribution
416 stars 248 forks source link

auto-stop-idle/autostop.py ignores open terminal sessions #90

Open rogalski opened 2 years ago

rogalski commented 2 years ago

Bug: When JupyterLab is open, but terminal sessions are used instead of notebook sessions, instance may get incorrectly shut down despite being actively used.

Debug:

  1. autostop.py uses sessions API: https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/blob/master/scripts/auto-stop-idle/autostop.py#L88
  2. Jupyter exposes info about notebooks and terminals under separate APIs:
  3. Activity on terminal never gets picked up by autostop.py which may result in premature shutdown

Caveats:

From my manual check 5 minutes ago:

sh-4.2$ curl -k https://localhost:8443/api/terminals/1
{"name": "1"}sh-4.2$
nethraui commented 1 year ago

Do you have any fix for this?

rvencu commented 1 year ago

bumping up this issue

takeru1205 commented 9 months ago

I opened PR that monitor notebook and terminal session! Please check this out!

dmlause commented 6 months ago

Thank you @takeru1205 - I left a comment on the PR.