squizlabs / PHP_CodeSniffer

PHP_CodeSniffer tokenizes PHP files and detects violations of a defined set of coding standards.
BSD 3-Clause "New" or "Revised" License
10.66k stars 1.48k forks source link

parallel processes that do nothing but need CPU #3144

Closed voku closed 2 years ago

voku commented 3 years ago

Describe the bug I don't know if this is a bug or not, but it looks strange:

Some processes seem to do nothing 0 wait4(18694, 0x7ffe6ddd6cbc, WNOHANG, NULL) = 0 (strace) but still need max CPU power (top).

Code sample Sorry, but the bug can't be shown in a small code sample, you need many files, so that one process is ready before other processes when we call PHP CodeSniffer via --parallel=X.

Custom ruleset

Custom ruleset ```xml The VDMG coding standard. error ./StandardVdmg/SniffsHelper/autoload.php 0 0 0 0 Only 1 @return annotation is allowed in a function comment Extra @param annotation Function has no return statement, but annotation @return is present @param annotation for parameter "%s" missing */tests/*\.php Please review this TODO comment: %s 3 warning Please review this FIXME comment: %s 5 warning 0 0 0 0 0 0 ```

To reproduce Run PHP CodeSniffer on a big code-base and then take a look at the processes via strace -p <PID> then you can see that there are processes that makes nothing "wait4" but still need ~ the same CPU power (see via (h)top) as the running processes.

Expected behavior If one process is done with his files, then the process does not need to use CPU power anymore, or?

Versions (please complete the following information):

Additional context

ps aux| grep php

lmoelle+ 10765  0.0  0.0   4632   928 pts/3    S+   08:19   0:00 sh -c php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar \         --runtime-set ignore_warnings_on_exit false \         --runtime-set ignore_errors_on_exit false \         --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ \         -s \         --colors \         --extensions=php \         --ignore="*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/" \         --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml \         --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 \         --parallel=4 \         --encoding=UTF-8 \         -p .     
lmoelle+ 10766 53.2  3.9 1004160 638796 pts/3  R+   08:19   6:27 php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar --runtime-set ignore_warnings_on_exit false --runtime-set ignore_errors_on_exit false --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ -s --colors --extensions=php --ignore=*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/ --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 --parallel=4 --encoding=UTF-8 -p .
lmoelle+ 10933 54.0  3.8 1010688 622948 pts/3  R+   08:19   6:28 php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar --runtime-set ignore_warnings_on_exit false --runtime-set ignore_errors_on_exit false --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ -s --colors --extensions=php --ignore=*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/ --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 --parallel=4 --encoding=UTF-8 -p .
lmoelle+ 10934 53.2  3.8 1017344 629268 pts/3  R+   08:19   6:23 php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar --runtime-set ignore_warnings_on_exit false --runtime-set ignore_errors_on_exit false --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ -s --colors --extensions=php --ignore=*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/ --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 --parallel=4 --encoding=UTF-8 -p .
lmoelle+ 10935 53.6  3.8 1010688 622672 pts/3  R+   08:19   6:26 php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar --runtime-set ignore_warnings_on_exit false --runtime-set ignore_errors_on_exit false --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ -s --colors --extensions=php --ignore=*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/ --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 --parallel=4 --encoding=UTF-8 -p .
lmoelle+ 10936 53.2  3.8 1012736 623604 pts/3  R+   08:19   6:23 php7.4 /home/lmoelleken/meerx/Framework/thirdparty/phpcs.phar --runtime-set ignore_warnings_on_exit false --runtime-set ignore_errors_on_exit false --runtime-set installed_paths /home/lmoelleken/meerx/App/scripts/githooks/StandardVdmg/,/home/lmoelleken/meerx/Framework/thirdparty/composer/phpcompatibility/php-compatibility/PHPCompatibility/ -s --colors --extensions=php --ignore=*.vue.php,/composer/,/vendor/,/archiv/,/thirdparty/,/MyFramework/generated/,/_support/_generated/,/tmp/,/LocalConfig/ --standard=/home/lmoelleken/meerx/App/scripts/githooks/phpcs_new.xml --cache=/home/lmoelleken/meerx/App/tmp/cache/phpcs_8f53409153411c054dff796307f63a82 --parallel=4 --encoding=UTF-8 -p .

sudo strace -p 10766

wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10933, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10934, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10935, 0x7ffef50ae57c, WNOHANG, NULL) = 0
wait4(10936, 0x7ffef50ae57c, WNOHANG, NULL) = 0
DavidGoodwin commented 3 years ago

I see this on PHP 7.4 ... for me, it's due to grpc.

The ini stuff here https://github.com/grpc/grpc/issues/20994#issuecomment-552624880 fixes it.

gsherwood commented 2 years ago

Cross-posting over here to see if anyone is able to test the fix in #3519 to see if it address this bug report.

DavidGoodwin commented 2 years ago

I left https://github.com/squizlabs/PHP_CodeSniffer/pull/3519#issuecomment-1040057274 on #3519 .... I'm not sure it's fixed.

voku commented 2 years ago

I left #3519 (comment) on #3519 .... I'm not sure it's fixed.

@DavidGoodwin

If you use "parallel" execution, then take a look at the different processes, e.g. via top or something like ps aux | grep php and then strace into one of these processes. The main process is still waiting for the child processes, but not every child process is waiting for all other child processes anymore. Can you confirm that?

DavidGoodwin commented 2 years ago

Running :

./vendor/bin/phpcbf --parallel=3 --standard=PSR12 --exclude=Generic.Files.LineLength ./lib/app ./tests

I see the parent stuck at :

strace: Process 1995388 attached
wait4(0, 

I managed to catch a child early enough on and got this -

newfstatat(AT_FDCWD, "/tmp/phpcs-childTLgqfy", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/tmp/phpcs-childTLgqfy", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
newfstatat(3, "", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0
lseek(3, 0, SEEK_CUR)                   = 0
write(3, "<?php\n $childOutput = array (\n  "..., 167) = 167
close(3)                                = 0
close(2)                                = 0
close(1)                                = 0
close(0)                                = 0
munmap(0x7f99a9800000, 2097152)         = 0
munmap(0x7f99a9c00000, 2097152)         = 0
munmap(0x7f99a9a00000, 2097152)         = 0
munmap(0x7f99a9600000, 2097152)         = 0
munmap(0x7f99a9200000, 2097152)         = 0
munmap(0x7f99ab96d000, 123888)          = 0
munmap(0x7f99ab7c6000, 1729800)         = 0
munmap(0x7f99ab98c000, 289184)          = 0
futex(0x7f99ac71a340, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f99ac719140, FUTEX_WAIT_PRIVATE, 0, NULL

And all children end up hanging on something like -

strace -p 1995396
strace: Process 1995396 attached
futex(0x7f969a746140, FUTEX_WAIT_PRIVATE, 0, NULL

If i run it with : php -d grpc.enable_fork_support=1 it behaves normally and doesn't block/wait forever.

PHP 7.4.27 (cli) (built: Dec 20 2021 21:28:51) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.27, Copyright (c), by Zend Technologies
    with Xdebug v3.1.2, Copyright (c) 2002-2021, by Derick Rethans

using the deb.sury.org repo with it's grpc module installed.

The same problem occurs with PHP8.0 (from the same source).

gsherwood commented 2 years ago

@voku has confirmed that this particular issue is resolved. The grpc issue is, I believe, something different and I don't think that's something I have control over. If anyone disagrees, please open a new report and we can track it there.