A proper fix for job control and parent process getting frozen?

tobixen commented 6 years ago

Suspending a child process causes side-effects for the parent sometimes (notably, bash job control - that's the only confirmed case I have so far, though I haven't done much research on this). Also, sometimes the parent process automatically gets suspended (notably, sudo).

Two work-arounds have been implemented so far. The first thing I did was to always resume parent the session process id and the group process id (I think the parent process id was not easily available from the scope where I did this), the second thing was to always stop the parent before the child if the parent process name was equal to "bash". In my upcoming commit, "sudo" has been added to this list.

I came to think that the proper fix for both those two issues may be to always freeze the parent process before suspending a child (possibly recursively, but never attempting on freezing pid 1 obviously). I need to think a bit and do some research before going this route.

Q-lds commented 3 years ago

Hello! I had two server crashes that resulted in the inability to ssh into the server and to use sudo.

I suspect that this is related to what you just wrote above, hence I am wondering if you have any recipe for what I should do after such occurrences!

In particular, what should I do to bring back sudo to its normal behaviour?

tobixen commented 3 years ago

Aww ... that's bad. Which version of thrash-protect did you use? Did you save on any logs?

Q-lds commented 3 years ago

I am currently using version 0.13.0

About the logs: yes, I think I should have most of them. What would you require? I am also planning to rerun the incriminated script over the weekend. Feel free to let me know whether I should check for anything in particular.

Also, maybe a bit of context regarding my use case: I have a VM with 250gb of ram and 25gb of swap memory, and the offending script was a jupyter notebook within a jupyterhub server.

tobixen commented 3 years ago

There was a bug in 0.13 that sometimes would cause frozen processes to be forgotten and never unfrozen in case of exceptions like running out of disk, maybe that's the problem you've encountered. 0.14.1 is the current version.

Except for that:

There shouldn't be any issue now with processes having the name "sudo", a workaround has been in place for several years specifically targetting processes with process name "sudo". If the process name is something different, then this issue will haunt you. (depending on a hard coded list of process names is obviously an ugly and unreliable workaround and not a proper fix).
Processes with process names like bash, ssh and sshd are (by default) whitelisted. Perhaps your command names comes up differently at your system.
I have some test code, but I don't remember if there is any test code specifically targeting cmdname and whitelisting/job control. Perhaps there is a bug in that part of the code.
Whitelisting does not mean the processes are completely safe from being suspended, in reality the results are simply weighted.
If your ssh server is particularly naughty or if the box is severely thrashed.

The log file at /var/log/thrash-protect.log contains the PID of all processes that have been frozen (suspended - I should probably add cmdname there as well) or unfrozen (resumed). It should also contain process information of all unfrozen processes.

What I'd like to know from the logs are:

Are there any pids that have been frozen, but not unfrozen?
Are there any pids that have been frozen and then waited for more than some few seconds before they got unfrozen?
Have any processes like ssh, sshd and sudo been frozen?

(are you sure thrash-protect is the culprit to the problems?)

Q-lds commented 3 years ago

Thanks for answering and helping out! Much appreciated.

So, I have checked the logs (attached at the end of the message). I think it is best for you to directly look at them :)

Before the logs, just a few remarks:

Regarding the severity of the thrashing ... well, we do many numerical simulations, so it is very easy for people to do linear algebra with matrices that scale as 2^n. So, when things go bad they tend to go bad quite fast. In this particular case, though, I did not see any OOM record from the syslog.
Indeed, having the command name saved could help when things don't work as planned.
Jupyterhub is started via a systemctl command (and the offending process was a jupyter notebook) so I guess Jupyterhub does run as root (indeed, many of the frozen pids are associated with root).
I see bash appearing in the thrash-protect logs, but I am unsure about what it means (see, for example, 2021-06-28 04:50:30 - unfrozen pid 92204 - u: user_3 CPU: 0.0% MEM: 0.0% CMD: bash - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,)] )
After the occurrence, I am usually left with many zombie processes. These often include some ssh commands (perhaps they are the visual studio you see in the logs?). I can usually use su but cannot directly use sudo. And usually, sometime after the memory issue, ssh also stops working.
I will try reproducing the same error I had a few days ago while using 0.14.1
I am currently using Ubuntu 20.04.2 LTS (not sure if this changes the command names)
I use byobu, an open-source text-based window manager and terminal multiplexer. Again, unsure if this may be causing any issue.
A simplified version of the function that cause the trashing is scipy.sparse.lil_matrix((2**n_spins, 2**n_spins)), where n_spins is an integer

Finally, I do not rule out that it may be something unrelated to thrash-protect! So, thanks for helping out!

logs

2021-06-28 04:31:07 - frozen pid 592116 - frozen list: [(592116,)]

2021-06-28 04:31:34 - unfrozen   pid 592116 - u:    user_7  CPU:  0.0%  MEM:  0.0%  CMD: /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub-singleuser --port=46191 --SingleUserNotebookApp.default_url=/lab - list: []

2021-06-28 04:33:52 - frozen pid 4033585 - frozen list: [(4033585,)]

2021-06-28 04:34:08 - unfrozen   pid 4033585 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: traefik traefik --providers.docker --providers.docker.constraints=Label(`traefik.constraint-label-stack`, `eos_dev.com`) --providers.docker.exposedbydefault=false --accesslog --log --api --api.insecure=true - list: []

2021-06-28 04:34:35 - frozen pid 125983 - frozen list: [(125983, 126828)]

2021-06-28 04:34:36 - unfrozen   pid 126828 - u:      user_1  CPU:  103%  MEM:  5.9%  CMD: python3 run.py - list: []

2021-06-28 04:34:36 - unfrozen   pid 125983 - u:      user_1  CPU:  0.0%  MEM:  0.0%  CMD: -bash - list: []

2021-06-28 04:36:26 - frozen pid 125983 - frozen list: [(125983,)]

2021-06-28 04:36:32 - unfrozen   pid 125983 - u:      user_1  CPU:  0.0%  MEM:  0.0%  CMD: -bash - list: []

2021-06-28 04:46:02 - frozen pid 131143 - frozen list: [(131143,)]

2021-06-28 04:46:04 - unfrozen   pid 131143 - u:   user_5  CPU:  109%  MEM:  0.0%  CMD: /.conda/envs/research/bin/python -m ipykernel_launcher -f /.local/share/jupyter/runtime/kernel-efe85807-65ce-4876-89b3-2c3a4d950f17.json - list: []

2021-06-28 04:46:10 - frozen pid 505691 - frozen list: [(505691,)]

2021-06-28 04:46:12 - frozen pid 845770 - frozen list: [(505691,), (845770, 890550)]

2021-06-28 04:46:37 - unfrozen   pid 890550 - u:        CPU:  9.2%  MEM:  0.0%  CMD: htop - list: [(505691,)]

2021-06-28 04:46:37 - unfrozen   pid 845770 - u:       user_10  CPU:  0.0%  MEM:  0.0%  CMD: /bin/bash - list: [(505691,)]

2021-06-28 04:46:37 - unfrozen   pid 505691 - u:      root  CPU:  0.0%  MEM:  0.f0%  CMD: node /usr/local/bin/configurable-http-proxy --ip --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0.0.1:8081/jupyter/hub/error - list: []

2021-06-28 04:48:45 - frozen pid 92255 - frozen list: [(92255,)]

2021-06-28 04:48:53 - unfrozen   pid 92255 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/vs/server/main.js --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.507ce72a4466fbb27b715c3722558bb15afa9f48.token - list: []

2021-06-28 04:49:52 - frozen pid 505691 - frozen list: [(505691,)]

2021-06-28 04:49:53 - unfrozen   pid 505691 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: node /usr/local/bin/configurable-http-proxy --ip --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0.0.1:8081/jupyter/hub/error - list: []

2021-06-28 04:50:03 - frozen pid 7413 - frozen list: [(7413,)]

2021-06-28 04:50:04 - frozen pid 1 - frozen list: [(7413,), (1,)]

2021-06-28 04:50:05 - frozen pid 132594 - frozen list: [(7413,), (1,), (132594,)]

2021-06-28 04:50:06 - unfrozen   pid 132594 - u:  systemd+  CPU:  0.0%  MEM:  0.0%  CMD: [sh] - list: [(7413,), (1,)]

2021-06-28 04:50:06 - unfrozen   pid     1 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /sbin/init - list: [(7413,)]

2021-06-28 04:50:07 - unfrozen   pid  7413 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd - list: []

2021-06-28 04:50:12 - frozen pid 132610 - frozen list: [(132610,)]

2021-06-28 04:50:13 - frozen pid 7413 - frozen list: [(132610,), (7413,)]

2021-06-28 04:50:14 - frozen pid 130264 - frozen list: [(132610,), (7413,), (130264, 130332)]

2021-06-28 04:50:14 - frozen pid 124403 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,)]

2021-06-28 04:50:15 - frozen pid 4033605 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,)]

2021-06-28 04:50:15 - frozen pid 4033558 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,)]

2021-06-28 04:50:17 - frozen pid 132612 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,)]

2021-06-28 04:50:17 - frozen pid 1996 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,)]

2021-06-28 04:50:18 - frozen pid 4035699 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,)]

2021-06-28 04:50:18 - frozen pid 4033493 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,)]

2021-06-28 04:50:18 - frozen pid 4034272 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886)]

2021-06-28 04:50:18 - frozen pid 4034242 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,)]

2021-06-28 04:50:19 - frozen pid 505658 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,)]

2021-06-28 04:50:21 - frozen pid 1668 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,)]

2021-06-28 04:50:21 - frozen pid 1630 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,)]

2021-06-28 04:50:21 - frozen pid 1630 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,)]

2021-06-28 04:50:21 - frozen pid 1623 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,)]

2021-06-28 04:50:22 - frozen pid 92298 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,)]

2021-06-28 04:50:22 - frozen pid 92255 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,)]

2021-06-28 04:50:22 - frozen pid 92204 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247)]

2021-06-28 04:50:23 - frozen pid 92338 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247), (92338,)]

2021-06-28 04:50:24 - frozen pid 4169381 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247), (92338,), (4169381,)]

2021-06-28 04:50:26 - frozen pid 1 - frozen list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247), (92338,), (4169381,), (1,)]

2021-06-28 04:50:29 - unfrozen   pid     1 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /sbin/init - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247), (92338,), (4169381,)]

2021-06-28 04:50:29 - unfrozen   pid 4169381 - u:    user_2  CPU:  0.0%  MEM:  0.2%  CMD: /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/node /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/out/vs/server/main.js --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.054a9295330880ed74ceaedda236253b4f39a335.token - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247), (92338,)]

2021-06-28 04:50:29 - unfrozen   pid 92338 - u:    user_3  CPU:  0.1%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/bootstrap-fork --type=extensionHost --uriTransformerPath=/.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/vs/server/uriTransformer.js - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,), (92204, 92247)]

2021-06-28 04:50:30 - unfrozen   pid 92247 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: sh /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.507ce72a4466fbb27b715c3722558bb15afa9f48.token - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,)]

2021-06-28 04:50:30 - unfrozen   pid 92204 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: bash - list: [(132610,), (7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,)]

2021-06-28 04:50:31 - unfrozen   pid 132610 - problem fetching process information - list: [(7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,), (92255,)]

2021-06-28 04:50:31 - unfrozen   pid 92255 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/vs/server/main.js --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.507ce72a4466fbb27b715c3722558bb15afa9f48.token - list: [(7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,), (92298,)]

2021-06-28 04:50:32 - unfrozen   pid 92298 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/bootstrap-fork --type=ptyHost - list: [(7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,), (1623,)]

2021-06-28 04:50:32 - unfrozen   pid  1623 - u:    user_4  CPU:  0.0%  MEM:  0.0%  CMD: sh /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.507ce72a4466fbb27b715c3722558bb15afa9f48.token - list: [(7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,), (1630,)]

2021-06-28 04:50:32 - unfrozen   pid  1630 - u:    user_4  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/vs/server/main.js --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --connection-secret /.vscode-server/.507ce72a4466fbb27b715c3722558bb15afa9f48.token - list: [(7413,), (130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,)]

2021-06-28 04:50:34 - unfrozen   pid  7413 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,), (1668,)]

2021-06-28 04:50:35 - unfrozen   pid  1668 - u:    user_4  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/bootstrap-fork --type=ptyHost - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,), (505658,)]

2021-06-28 04:50:35 - unfrozen   pid 505658 - u:      root  CPU:  0.1%  MEM:  0.0%  CMD: /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886), (4034242,)]

2021-06-28 04:50:36 - unfrozen   pid 4034242 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id f1f9a3b71e565ff97f2f644e1bc3fbc66f5c7d07f73b944bee6a6cdfe9b5422f -address /run/containerd/containerd.sock - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,), (4034272, 4034886)]

2021-06-28 04:50:36 - unfrozen   pid 4034886 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/local/bin/python3 /usr/local/bin/celery worker -A app.worker -l info -Q main-queue -c 1 - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,)]

2021-06-28 04:50:36 - unfrozen   pid 4034272 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: bash /worker-start.sh - list: [(130264, 130332), (124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,)]

2021-06-28 04:50:36 - unfrozen   pid 130332 - u:      user_1  CPU: 1201%  MEM:  0.1%  CMD: python3 run.py - list: [(124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,)]

2021-06-28 04:50:36 - unfrozen   pid 130264 - u:      user_1  CPU:  0.0%  MEM:  0.0%  CMD: -bash - list: [(124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,), (4033493,)]

2021-06-28 04:50:37 - unfrozen   pid 4033493 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id 37ccf575c12b40af3cf10776d27cef5a547b6c65ff091d1f3202e97e13d0185a -address /run/containerd/containerd.sock - list: [(124403,), (4033605,), (4033558,), (132612,), (1996,), (4035699,)]

2021-06-28 04:50:38 - unfrozen   pid 4035699 - problem fetching process information - list: [(124403,), (4033605,), (4033558,), (132612,), (1996,)]

2021-06-28 04:50:41 - unfrozen   pid  1996 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/python3 /usr/bin/google_clock_skew_daemon - list: [(124403,), (4033605,), (4033558,), (132612,)]

2021-06-28 04:50:41 - unfrozen   pid 132612 - problem fetching process information - list: [(124403,), (4033605,), (4033558,)]

2021-06-28 04:50:41 - unfrozen   pid 124403 - u:    user_3  CPU:  122%  MEM:  0.0%  CMD: /.vscode-server/data/User/globalStorage/tabnine.tabnine-vscode/binaries/3.5.12/x86_64-unknown-linux-musl/WD-TabNine --client vscode --client-metadata clientVersion=1.57.1 pluginVersion=3.4.10 t9-vscode-AutoImportEnabled=true t9-vscode-TSAutoImportEnabled=true t9-vscode-JSAutoImportEnabled=true vscode-telemetry-enabled=true vscode-remote=true vscode-remote-name=ssh-remote vscode-extension-kind=2 vscode-theme-name=Default Dark+ vscode-theme-kind=Dark vscode-machine-id=b86070203c7ca3646f8df5a245d5714e0509b898f4a9148ebdc752948d64e6d3 vscode-is-new-app-install=false vscode-session-id=0823d41d-650b-4f81-94a0-09b8975b03411624846462036 vscode-language=en vscode-app-name=Visual Studio Code vscode-beta-channel-enabled=false vscode-status-customization=unknown ide-restart-counter=1 - list: [(4033605,), (4033558,)]

2021-06-28 04:50:42 - unfrozen   pid 4033558 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id b85d3642ecbdea2ea5dc3c895207994dd785c428dd4f894228f84ed6f35e8250 -address /run/containerd/containerd.sock - list: [(4033605,)]

2021-06-28 04:50:42 - unfrozen   pid 4033605 - u:    ubuntu  CPU:  0.0%  MEM:  0.0%  CMD: /usr/local/bin/python /usr/local/bin/flower --broker=amqp://guest@queue:5672// - list: []

2021-06-28 04:50:53 - frozen pid 4033605 - frozen list: [(4033605,)]

2021-06-28 04:50:53 - unfrozen   pid 4033605 - u:    ubuntu  CPU:  0.0%  MEM:  0.0%  CMD: /usr/local/bin/python /usr/local/bin/flower --broker=amqp://guest@queue:5672// - list: []

2021-06-28 04:50:55 - frozen pid 1668 - frozen list: [(1668,)]

2021-06-28 04:50:56 - frozen pid 1668 - frozen list: [(1668,)]

2021-06-28 04:50:57 - frozen pid 1995 - frozen list: [(1668,), (1995,)]

2021-06-28 04:50:57 - frozen pid 132759 - frozen list: [(1668,), (1995,), (132759,)]

2021-06-28 04:50:58 - frozen pid 112342 - frozen list: [(1668,), (1995,), (132759,), (112342,)]

2021-06-28 04:50:59 - unfrozen   pid 112342 - u:      user_1  CPU:  7.4%  MEM:  0.0%  CMD: /.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/node /.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/out/bootstrap-fork --type=extensionHost --uriTransformerPath=/.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/out/vs/server/uriTransformer.js - list: [(1668,), (1995,), (132759,)]

2021-06-28 04:51:00 - frozen pid 112342 - frozen list: [(1668,), (1995,), (132759,), (112342,)]

2021-06-28 04:51:01 - frozen pid 1564788 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,)]

2021-06-28 04:51:01 - frozen pid 1630 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,)]

2021-06-28 04:51:01 - frozen pid 1630 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,)]

2021-06-28 04:51:02 - frozen pid 4034783 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,)]

2021-06-28 04:51:02 - frozen pid 132774 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,)]

2021-06-28 04:51:04 - frozen pid 92338 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,)]

2021-06-28 04:51:05 - frozen pid 4033605 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,)]

2021-06-28 04:51:05 - frozen pid 92255 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,)]

2021-06-28 04:51:05 - frozen pid 4033558 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,)]

2021-06-28 04:51:05 - frozen pid 7413 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,)]

2021-06-28 04:51:05 - frozen pid 7413 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,)]

2021-06-28 04:51:05 - frozen pid 504010 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,)]

2021-06-28 04:51:06 - frozen pid 508110 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,)]

2021-06-28 04:51:06 - frozen pid 598 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,)]

2021-06-28 04:51:07 - frozen pid 2285419 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,)]

2021-06-28 04:51:07 - frozen pid 1 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,)]

2021-06-28 04:51:07 - frozen pid 10796 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,)]

2021-06-28 04:51:07 - frozen pid 10796 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,)]

2021-06-28 04:51:09 - frozen pid 12790 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,)]

2021-06-28 04:51:09 - frozen pid 505691 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,)]

2021-06-28 04:51:09 - frozen pid 4034780 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,)]

2021-06-28 04:51:09 - frozen pid 104376 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,)]

2021-06-28 04:51:10 - frozen pid 104369 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,)]

2021-06-28 04:51:12 - frozen pid 93533 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,)]

2021-06-28 04:51:12 - frozen pid 93533 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,)]

2021-06-28 04:51:13 - frozen pid 504294 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,)]

2021-06-28 04:51:13 - frozen pid 93254 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,)]

2021-06-28 04:51:13 - frozen pid 93254 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,)]

2021-06-28 04:51:14 - frozen pid 93254 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,)]

2021-06-28 04:51:15 - frozen pid 124403 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,)]

2021-06-28 04:51:15 - frozen pid 92298 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,)]

2021-06-28 04:51:16 - frozen pid 1044295 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023)]

2021-06-28 04:51:16 - frozen pid 92497 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,)]

2021-06-28 04:51:16 - frozen pid 92497 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,)]

2021-06-28 04:51:16 - frozen pid 4169419 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,)]

2021-06-28 04:51:17 - frozen pid 4169381 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,)]

2021-06-28 04:51:17 - frozen pid 3531937 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,)]

2021-06-28 04:51:18 - frozen pid 3531975 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,)]

2021-06-28 04:51:18 - frozen pid 3531930 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,)]

2021-06-28 04:51:19 - frozen pid 2048796 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,)]

2021-06-28 04:51:19 - frozen pid 4033585 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,)]

2021-06-28 04:51:19 - frozen pid 4169703 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,)]

2021-06-28 04:51:19 - frozen pid 1428 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,)]

2021-06-28 04:51:19 - frozen pid 1429 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,)]

2021-06-28 04:51:19 - frozen pid 4169463 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,)]

2021-06-28 04:51:19 - frozen pid 1427 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,)]

2021-06-28 04:51:20 - frozen pid 1427 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,)]

2021-06-28 04:51:20 - frozen pid 92321 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,)]

2021-06-28 04:51:20 - frozen pid 11218 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,)]

2021-06-28 04:51:21 - frozen pid 4169436 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,)]

2021-06-28 04:51:22 - frozen pid 572880 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,)]

2021-06-28 04:51:22 - frozen pid 572880 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,)]

2021-06-28 04:51:23 - frozen pid 4033373 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,)]

2021-06-28 04:51:23 - frozen pid 4033383 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,)]

2021-06-28 04:51:23 - frozen pid 4033470 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,), (4033470,)]

2021-06-28 04:51:24 - frozen pid 505658 - frozen list: [(1668,), (1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,), (4033470,), (505658,)]

2021-06-28 04:51:29 - unfrozen   pid  1668 - u:    user_4  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/bootstrap-fork --type=ptyHost - list: [(1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,), (4033470,), (505658,)]

2021-06-28 04:51:31 - unfrozen   pid 505658 - u:      root  CPU:  0.1%  MEM:  0.0%  CMD: /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py - list: [(1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,), (4033470,)]

2021-06-28 04:51:32 - unfrozen   pid 4033470 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5435566fc645f99cb04eb505e480b777b949252420ec6983e0dea4a2a9292c6d -address /run/containerd/containerd.sock - list: [(1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,), (4033383,)]

2021-06-28 04:51:33 - unfrozen   pid 4033383 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id 639f56b394835966bb4fe802fc2f3b6390fd2a8c5dec2f1ffd19f26b73a5d4cd -address /run/containerd/containerd.sock - list: [(1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,), (4033373,)]

2021-06-28 04:51:34 - unfrozen   pid 4033373 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5040bb698e3214322b979bc60f7db09913db8746da584e3c932a36eb7b6dc82a -address /run/containerd/containerd.sock - list: [(1995,), (132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,)]

2021-06-28 04:51:35 - unfrozen   pid  1995 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: /usr/bin/python3 /usr/bin/google_accounts_daemon - list: [(132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,), (572880,)]

2021-06-28 04:51:35 - unfrozen   pid 572880 - u:      user_1  CPU:  0.0%  MEM:  0.0%  CMD: /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub-singleuser --port=59547 --SingleUserNotebookApp.default_url=/lab - list: [(132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,), (4169436,)]

2021-06-28 04:51:36 - unfrozen   pid 4169436 - u:    user_2  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/node /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/out/bootstrap-fork --type=watcherService - list: [(132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,), (11218,)]

2021-06-28 04:51:36 - unfrozen   pid 11218 - u:  postgres  CPU:  0.0%  MEM:  0.0%  CMD: postgres: 13/main: background writer - list: [(132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,), (92321,)]

2021-06-28 04:51:36 - unfrozen   pid 92321 - u:    user_3  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/node /.vscode-server/bin/507ce72a4466fbb27b715c3722558bb15afa9f48/out/bootstrap-fork --type=watcherService - list: [(132759,), (112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,)]

2021-06-28 04:51:37 - unfrozen   pid 132759 - problem fetching process information - list: [(112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,), (1427,)]

2021-06-28 04:51:38 - unfrozen   pid  1427 - u:      root  CPU:  0.0%  MEM:  0.0%  CMD: nginx: master process /usr/sbin/nginx -g daemon on; master_process on; - list: [(112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,), (4169463,)]

2021-06-28 04:51:38 - unfrozen   pid 4169463 - u:    user_2  CPU:  0.0%  MEM:  0.0%  CMD: /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/node /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/out/bootstrap-fork --type=extensionHost --uriTransformerPath=/.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/out/vs/server/uriTransformer.js - list: [(112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,), (1429,)]

2021-06-28 04:51:46 - unfrozen   pid  1429 - u:  www-data  CPU:  0.0%  MEM:  0.0%  CMD: nginx: worker process - list: [(112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,), (1428,)]

2021-06-28 04:51:46 - unfrozen   pid  1428 - u:  www-data  CPU:  0.0%  MEM:  0.0%  CMD: nginx: worker process - list: [(112342,), (1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,)]

2021-06-28 04:51:46 - unfrozen   pid 112342 - u:      user_1  CPU:  7.4%  MEM:  0.0%  CMD: /.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/node /.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/out/bootstrap-fork --type=extensionHost --uriTransformerPath=/.vscode-server-insiders/bin/49e1f9919880c1c96004f9b6ad91bcee7a94a1bd/out/vs/server/uriTransformer.js - list: [(1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,), (4169703,)]

2021-06-28 04:51:54 - unfrozen   pid 4169703 - u:    user_2  CPU:  0.0%  MEM:  0.2%  CMD: /.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/node /.vscode-server/extensions/ms-python.vscode-pylance-
2021.6.2/dist/server.bundle.js --cancellationReceive=file:fe92b27b5479e22e1da4293e5bb60c87214ce4540d --node-ipc --clientProcessId=4169463 - list: [(1564788,), (1630,), (4034783,), (132774,), (92338,), (4033605,), (92255,), (4033558,), (7413,), (504010,), (508110,), (598,), (2285419,), (1,), (10796,), (12790,), (505691,), (4034780,), (104376,), (104369,), (93533,), (504294,), (93254,), (124403,), (92298,), (1044295, 2757023), (92497,), (4169419,), (4169381,), (3531937,), (3531975,), (3531930,), (2048796,), (4033585,)]

tobixen commented 3 years ago

Incidentally, I experienced hard thrashing on a VM today, had to resort to a hard reboot even if thrash-protect was running. I'm doing some development on two hobby projects, and haven't bothered making a service yet, so I did a while [ 1 ] ; do python myproject.py ; sleep 10 ; done in a tmux session. When the box went offline, it had a lot of suspended python processes.

I'm not sure if it's related to the issue you're experiencing, but I will prioritize research into this one.

Q-lds commented 3 years ago

Sure, fair enough.

I tried to crash the machine twice yesterday using version 0.14.1, and in both cases, thrash-protect worked as intended. However, I was the only one running processes, and perhaps that played a role.

tobixen commented 3 years ago

I managed to thrash my laptop so badly yesterday that it was needed with a hard reboot ... so indeed, I do have some work to do. I should obviously spin up some VMs so that I can do less disruptive testing. I'm not sure when I will get time for it, I'm quite busy those days, but it's definitively a priority.

tobixen commented 3 years ago

So far I've found this:

Running some simple while [ 1 ] ; do sleep 120 ; done in bash (or eventually something like for x in 1 2 3 4 ; do tests/thrash-bot.py ; done), and then suspending the sleep process, will cause a new sleep process to be spawned. Resume the old sleep process and it will appear as a "background job" in bash.
suspend the bash process first and then the sleep process, and then resume the sleep process before resuming the bash process, and things will work out as intended
thrash_protect seems to do things in the wrong order by now, it suspends the sleep process first and then the bash process - so this will cause multiple parallell thrash-bot.py-processes to be spawned up. This is very weird, I'm quite sure that this has worked as intended at some point - and there is even some unit tests covering this.
If the parent process is suspended, and a child process exits, the child process will hang around in zombie status until the parent process resumes and can read the exit code from the child process. This is normal, nothing to worry about.

The next step for me now is obviously to try to cover the error in the third point in some unit test. It seems like I will be fairly busy for the rest of the day and probably tomorrow as well, so I'm not sure when I will be able to come back to this one.

I should also:

Make sure I can run multiple instances of thrash_bot.py on some VM, and still be able to log in through ssh.
Make sure I can run multiple instances of thrash_bot.py without losing control totally of my laptop.
cmdname should be relatively easily available in the code, make sure that it's logged when the process is frozen.

(It would actually make sense to raise all of this as separated issues - the original issue is to consider if this "special logic" on suspending the parent before suspending the child should be made generic and apply to all processes, and not only if the parent process has the name "bash" or "sudo". Indeed, it seems important for me - we have sort of cloned bash at work and given it a new name "tabascho" - bash with some extra capabilities, so that tab-completion will work for admin workers even on directories where said admin user does not have read permissions)

tobixen commented 3 years ago

The code looks correct. I noticed this:

2021-07-04 09:21:38 - frozen pid 215638 - frozen list: [(215638, 217559)]
2021-07-04 09:21:38 - frozen pid 217559 - frozen list: [(215638, 217559)]
2021-07-04 09:21:38 - frozen pid 492 - frozen list: [(215638, 217559), (492,)]
2021-07-04 09:21:40 - unfrozen   pid 217559 - u:    tobias  CPU: 81.9%  MEM: 67.4%  CMD: /usr/bin/python3 ./thrash-bot.py - list: [(492,)]
2021-07-04 09:21:40 - unfrozen   pid 215638 - u:    tobias  CPU:  0.0%  MEM:  0.0%  CMD: /bin/bash - list: [(492,)]

This is correct. bash is suspended before thrash-bot, and thrash-bot is resumed before bash. But then the next lines looks like this:

2021-07-04 09:21:40 - frozen pid 217559 - frozen list: [(492,), [217559, 215638]]
2021-07-04 09:21:40 - frozen pid 215638 - frozen list: [(492,), [217559, 215638]]
2021-07-04 09:21:41 - frozen (some more pids)
2021-07-04 09:21:42 - unfrozen  (some more stuff)
2021-07-04 09:21:42 - unfrozen   pid 215638 - u:    tobias  CPU:  0.0%  MEM:  0.0%  CMD: /bin/bash - list: [(492,)]
2021-07-04 09:21:43 - unfrozen   pid 217559 - u:    tobias  CPU: 65.4%  MEM: 68.4%  CMD: /usr/bin/python3 ./thrash-bot.py - list: [(492,)]

I have found the issue, unfreeze_something() makes a list of what it unfreezes (and that list is reversed, compared to what was frozen) and returns it, it gets stored it so that the last frozen process quickly can be frozen again should the system start thrashing again immediately. This list should obviously not be stored in the reversed order.

tobixen commented 3 years ago

I didn't write any unit tests so far - should do that, too. Anyway, I'm way on overtime, was planning to drop the laptop and deal with other things an hour ago already :-)

tobixen / thrash-protect

A proper fix for job control and parent process getting frozen? #12