502 Bad Gateway sometimes

[koekaverna]

I added extension to docker image It cause 502 Error sometimes.

php_1 | [10-Aug-2021 14:16:44] WARNING: [pool www] child 8 exited on signal 11 (SIGSEGV - core dumped) after 127.559666 seconds from start php_1 | [10-Aug-2021 14:16:44] NOTICE: [pool www] child 62 started http_1 | 2021/08/10 14:16:44 [error] 25#25: *3 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.48.1, server: localhost, request: "POST /auth/token?w HTTP/1.1", upstream: "fastcgi://192.168.48.4:9000", host: "127.0.0.2"

I tried it with php8 and php7.4

[longxinH]

Is the extension compiled in docker? Or is it mapped into docker after the host is compiled?

[koekaverna]

Is the extension compiled in docker? Or is it mapped into docker after the host is compiled?

Complied in Docker. I Tried with php:8.0-fpm-alpine and php:7.4.16-fpm-alpine3.13

ENV XHPROF_VERSION 2.3.4
RUN set -x \
    && mkdir -p /usr/src/php/ext/xhprof \
    && curl "https://pecl.php.net/get/xhprof/${XHPROF_VERSION}" \
        | tar xvz --directory=/usr/src/php/ext/xhprof --strip=2 xhprof-${XHPROF_VERSION}/extension \
    && docker-php-ext-install -j$(nproc) xhprof \
    && rm -rf /usr/src/php/ext/xhprof

build.log

[longxinH]

xdebug and xhprof will conflict and cannot be loaded at the same time

[koekaverna]

@longxinH thank you for the answer. Without xdebug it woks with no problem.

[koekaverna]

After testing extension for few days we realised that the bug still happens. And huge amount of requests terminated by sigserv 11 (25-50%). It happen on production enviroment without xdebug extension and with enabled opcache.

[longxinH]

Can you provide GDB report?

[koekaverna]

Can you provide GDB report?

Is it enough the backtrace? Should I provide something else?

(gdb) bt
#0  0x00007f24c99d6d13 in hp_init_profiler_state ()
   from /usr/local/lib/php/extensions/no-debug-non-zts-20190902/xhprof.so
#1  0x00007f24c99d6df0 in hp_begin.part () from /usr/local/lib/php/extensions/no-debug-non-zts-20190902/xhprof.so
#2  0x00007f24c99d70af in zif_xhprof_enable ()
   from /usr/local/lib/php/extensions/no-debug-non-zts-20190902/xhprof.so
#3  0x000055b2e6893190 in execute_ex ()
#4  0x000055b2e6894e76 in execute_ex ()
#5  0x000055b2e6894e76 in execute_ex ()
#6  0x000055b2e6894e76 in execute_ex ()
#7  0x000055b2e6892e45 in execute_ex ()
#8  0x000055b2e68760dc in ?? ()
#9  0x000055b2e688dcea in execute_ex ()
#10 0x000055b2e68962fd in zend_execute ()
#11 0x000055b2e6810254 in zend_execute_scripts ()
#12 0x000055b2e67afa08 in php_execute_script ()
#13 0x000055b2e65520ab in ?? ()
#14 0x00007f24cc84fa03 in ?? () from /lib/ld-musl-x86_64.so.1
#15 0x00007f24cc84f9dc in ?? () from /lib/ld-musl-x86_64.so.1
#16 0x00007fff266a3db0 in ?? ()
#17 0x0000000000000000 in ?? ()

[longxinH]

I tested it based on php:8.0.0-fpm-alpine3.12 everything is normal, can you provide a detailed test environment?

[koekaverna]

I tryed to make reproducer repository, but have no success on it. It work fine with clean symfony 5.3 installation, so probably there is some kind of conflict in extensions. I will try to find something usefull in core dump reports.

[eimkua1]

I also have this issue. It started from version 2.3.* and everything is fine with 2.2.3

[longxinH]

@eimkua1 Can you provide the relevant environmental information and reproduction code?

[koekaverna]

@longxinH Finally I found the bottleneck and was able to create a reproduction repo. You can find it here https://github.com/koekaverna/XhprofTestProject

The problem occurs when I use php-profiler library with flush option which is default. And I think the only way to use it in production mode. This flush cause 502 Bad Gateway error. But when I use it with tideways-xhprof it works just fine.

I make some test deployment, you can take a look on it: https://github.com/koekaverna/XhprofTestProject/actions And also there is an explanation how to set up and reproduce the problem in README

Hope it will help you to find out the problem.

[longxinH]

@koekaverna It has been fixed, try the latest master branch

[longxinH]

https://pecl.php.net/package/xhprof/2.3.5

[koekaverna]

It's works fine and with X debug too.