Scheduling fails jupyter running on container if a container restart

[loustler]

I saw scheduling failed when container restarts by any reasons.

I think many companies are running jupyter on container system(Kubernetes, AWS ECS and etc), in this system, we already know application restarts anytime. If application is restarting, changing runtime environment like environment variable and IPs . In 0.1.5, that situations not covers. Because adds environment variables when add job, jupyter is restarted, container environment changed, so saved environments variable is invalid. This make trouble on some container system like AWS ECS.

I suggest the flag to choose save environment variable to cron config or not. The flag is environment variable like JUPYTER_LAB_SCHEDULER_SAVE_SYSTEM_ENV_VARS. Or needs sync environment variables when jupyter starts up.

[shubhamkanwal]

Hi @loustler , i am also running jupyterhub on EKS cluster. Using same scipy docker file as mentioned in readme. However, getting error" There was an error submitting job".

I have also tried 2 solutions mentioned for this error, but it's not working. Please help!

[loustler]

@shubhamkanwal I just migrated to EKS from ECS.

I'm found same problem with you.

I just solved this problem.

I'm set securityContext property into pod spec.

spec:
  securityContext: 
    fsGroup: 100 # this is group id in jupyter official docker. Check your GID
  containers:
  - securityContext:
      allowPrivilegeEscalation: false
      runAsUser: 0 # run container as root user for starting crontab service

The jupyter pod log said like permission denied: /var/spool/cron/crontabs, i guess.

so I was find out file permission, and change pod configuration, fixed it.

For further information, see k8s docs

Hope this helps!

[jovanni-hernandez]

@loustler thanks for sharing your fix, I'll get it added to the docs