CHTC / dask-chtc

Spawn Dask workers in the CHTC HTCondor pool
https://dask-chtc.readthedocs.io
3 stars 3 forks source link

Add a --force option to jupyter stop #30

Closed JoshKarpel closed 4 years ago

JoshKarpel commented 4 years ago

This PR resolves #29 by encapsulating the manual workaround we found while troubleshooting, as described in https://github.com/JoshKarpel/dask-chtc/issues/29#issuecomment-661954094 .

A force stop looks for the notebook server process tree and sends it a kill signal. It looks like this:

$ dask-chtc jupyter stop --force
000 (16453.000.000) 2020-07-21 11:58:25 Job submitted from host: <10.0.1.43:40415?addrs=10.0.1.43-40415+[2600-6c44-1180-1661-99fa-fc04-10e3-fd8d]-40415&alias=JKARPEL&noUDP&sock=schedd_20423_5f31>
001 (16453.000.000) 2020-07-21 11:58:27 Job executing on host: <10.0.1.43:40415?addrs=10.0.1.43-40415+[2600-6c44-1180-1661-99fa-fc04-10e3-fd8d]-40415&alias=JKARPEL&noUDP&sock=starter_20464_7d39_11>
005 (16453.000.000) 2020-07-21 11:58:30 Job terminated.
    (0) Abnormal termination (signal 9)
    (0) No core file
        Usr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage
        Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
        Usr 0 00:00:00, Sys 0 00:00:00  -  Total Remote Usage
        Usr 0 00:00:00, Sys 0 00:00:00  -  Total Local Usage
    0  -  Run Bytes Sent By Job
    0  -  Run Bytes Received By Job
    0  -  Total Bytes Sent By Job
    0  -  Total Bytes Received By Job

This is a low-level hack that we will probably need to revisit later.