Q: curio kernel thread priority

[Andrei-Pozolotin]

@dabeaz David:

1. given all your deep GIL hacking experience https://www.youtube.com/watch?v=Obt-vMVdM8s

2. say, an application is running curio in main thread as an orchestration framework only (no heavy i/o, no heavy math), then pushing bytes via asyncio, and doing heavy math via threading - all in separate threads,

3. do you think curio kernel thread priority elevation approaches such as pthread_setschedparam() for Linux and SetThreadPriority() for Windows can prevent main thread starvation, i.e. protection against both heavy i/o threads and heavy math threads?

[dabeaz]

Short answer: Without some experiments and data, I have no idea.

My gut feeling is that changing the curio kernel thread priority is going to have little effect (if any) on how Python actually executes threads due to issues in how the GIL is implemented. For example, there is an inherent delay in acquiring the GIL from any CPU-bound thread that is present independently of priorities.

Of course, if the heavy math threads were to run with the GIL released (i.e., involving C extensions, etc.), it might be a different story.

[Andrei-Pozolotin]

ok, so far:

1. based on: http://www.yonch.com/tech/82-linux-thread-priority https://gitlab.com/guystreeter/python-hwloc/blob/master/linuxsched.pyx https://github.com/python-llfuse/python-llfuse/blob/master/Include/pthread.pxd

2. I can set priority to RT/99: https://github.com/random-python/data_pipe/blob/master/src/test/data_pipe_test/verify_priority.py https://github.com/random-python/data_pipe/blob/master/src/main/data_pipe/native_system.pxd https://github.com/random-python/data_pipe/blob/master/src/main/data_pipe/native_module.pyx#L13

3. but that requires root do you know if this can be root-less?

[dabeaz]

Messing around with thread priorities and scheduling has historically always required root access if your goal is to increase the priority beyond the default setting. A user can usually decrease their scheduling priority however. So, that might be a different option (decrease the scheduling priority of CPU-intensive threads and run everything else at normal priority).

[Andrei-Pozolotin]

great idea, will try that.

[Andrei-Pozolotin]

1. look: secret santa already committed os.setpriority and os.getpriority 10 years back: https://bugs.python.org/issue10784 which is present but not documented: Modules/posixmodule.c

2. meanwhile, using os.nice, which is documented the following experiment: https://gist.github.com/Andrei-Pozolotin/14187e1c61e0fa56866077c5acfd92ce where 1 master is running at normal nice priority and 9 worker are running at minimal nice priority, produces "negative result": all 10 thread are performing at even 10%, despite top/htop confirm proper nice setting per thread

3. do you see an error in the experiment's code? if not, where/how GIL manages to mess up the os scheduler?

[Andrei-Pozolotin]

1. porting from threading to multiprocessing, next experiment: https://gist.github.com/Andrei-Pozolotin/3e548c640000982de98f0b701c7e0b21 again top/htop shows proper nice level, though unlike threading, now each process shows 99.9% CPU% (tested on 32 core box, so that seems possible)

2. the experiment produces the same "negative result", meaning it shows even 10% performance ratio between processes which can be interpreted as:

   • nice does not cut it?
   • can blame GIL no more?
   • error in the experiment's code?

3. any insight is much appreciated