bootstrap spawns dozens of python processes simultaneously, enough to freeze a computer for a few seconds

sagemath / sage

Main repository of SageMath

https://www.sagemath.org

Other

1.37k stars 466 forks source link

bootstrap spawns dozens of python processes simultaneously, enough to freeze a computer for a few seconds #38753

Open cyrilbouvier opened 1 week ago

cyrilbouvier commented 1 week ago

Steps To Reproduce

Run make

Expected Behavior

My machine should still be usable while SageMath is building

Actual Behavior

At the beginning of the make command, my machine becomes unresponsive for a few seconds. If I run top while running, I can clearly see dozens and dozens of python processes all appearing at the same time.

Additional Information

I think this come from the last line of src/doc/bootstrap :

sage-package list --has-file SPKG.rst | OUTPUT_DIR=$OUTPUT_DIR OUTPUT_RST=1 xargs -P 99 -n 1 sage-spkg-info

which will call sage-spkg-info 99 times simultaneously. I think it should be -P 1 or, if possible, the user could overwrite this value (with 1 as default).

Environment

OS: Debian 11
Sage Version: 10.5.beta6

Checklist

[X] I have searched the existing issues for a bug report that matches the one I want to file, without success.
[X] I have read the documentation and troubleshoot guide

fchapoton commented 3 days ago

this was done in #37104

orlitzky commented 3 days ago

this was done in #37104

Actually, it used to be even slower. xargs -P 0 means run them all at the same time. Now at least you are limited to 99 processes!

orlitzky commented 3 days ago

I was a little surprised that parallel processing has such an effect on this, since it's all I/O.

With -P4:

real    2m38.781s
user    9m0.266s
sys 1m19.429s

With -P1:

real    7m25.291s
user    6m19.946s
sys 1m5.246s

With -P99:

real    3m5.799s
user    10m51.435s
sys 1m19.453s

So -P<nprocs> is probably the "right" number. It still wastes a lot of CPU time, but it saves real time. What is really outrageous though is that I am spending three minutes copying 1.8 megabytes of text files into a directory.