openswoole / ext-openswoole

Programmatic server for PHP with async IO, coroutines and fibers
https://openswoole.com
Apache License 2.0
807 stars 50 forks source link

Crashes with coroutines and curl 8.10.1 #366

Open hydrapolic opened 3 weeks ago

hydrapolic commented 3 weeks ago

Since 1.10.2024, our application started to crash:

app-1      | [2024-10-04 06:52:16 $119.0]       WARNING Server::check_worker_exit_status(): worker(pid=711, id=0) abnormal exit, status=0, signal=11
app-1      | A bug occurred in OpenSwoole-v22.1.2, please report it.
app-1      | Please submit bug report at:
app-1      | >> https://github.com/openswoole/swoole-src/issues 
app-1      | 
app-1      | OS: Linux 6.6.51-gentoo #1 SMP PREEMPT_DYNAMIC Fri Sep 13 23:06:02 UTC 2024 x86_64
app-1      | GCC_VERSION: 13.2.1 20240309
app-1      | OPENSSL_VERSION: OpenSSL 3.3.2 3 Sep 2024
app-1      | PHP_VERSION : 8.3.11

From our Dockerfile:

FROM php:8.3.11-alpine3.20

RUN apk upgrade --no-cache \
    && apk add --no-cache \
    curl-dev

# Install extensions
RUN pecl install -D 'enable-openssl="yes" enable-hook-curl="yes"' openswoole-22.1.2

There is a lot of inter-service communication between the microservices via Guzzlehttp and it only happens with coroutines enabled.

That time, curl was upgraded in Alpine 3.20: https://gitlab.alpinelinux.org/alpine/aports/-/commit/e17ad295e0810e938f9c4d61c21e44eaf1a59b51 https://gitlab.alpinelinux.org/alpine/aports/-/commit/badfda8e31a0b6e87e78237d7ee4455b35cd4c10

After reverting back to Alpine 3.19 it works fine.

Alpine 3.19: curl 8.9.1 Alpine 3.20: curl 8.10.1

Maybe related: https://github.com/swoole/swoole-src/issues/5500 https://github.com/curl/curl/issues/15127

php --ri openswoole

openswoole

Open Swoole => enabled
Author => Open Swoole Group <hello@openswoole.com>
Version => 22.1.2
Built => Oct  4 2024 06:56:46
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 3.1.7 3 Sep 2024
dtls => enabled
hook-curl => enabled
zlib => 1.3.1
brotli => E16781312/D16781312
mutex_timedlock => enabled
pthread_barrier => enabled

Directive => Local Value => Master Value
openswoole.enable_coroutine => On => On
openswoole.enable_preemptive_scheduler => Off => Off
openswoole.display_errors => On => On
openswoole.unixsock_buffer_size => 8388608 => 8388608

uname -a

Linux 5ff868dd84d7 6.6.51-gentoo #1 SMP PREEMPT_DYNAMIC Fri Sep 13 23:06:02 UTC 2024 x86_64 Linux

php -v

PHP 8.3.11 (cli) (built: Sep  7 2024 01:13:27) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.11, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.11, Copyright (c), by Zend Technologies

gcc -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/13.2.1/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-13-20231014/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-l
inux-musl --target=x86_64-alpine-linux-musl --enable-checking=release --disable-cet --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers 
--enable-__cxa_atexit --enable-default-pie --enable-default-ssp --enable-languages=c,c++,d,objc,go,fortran,ada --enable-link-serialization=2 --enable-linker-build-id --disable-libssp --disable-libsa
nitizer --enable-shared --enable-threads --enable-tls --with-bugurl=https://gitlab.alpinelinux.org/alpine/aports/-/issues --with-system-zlib --with-linker-hash-style=gnu --with-pkgversion='Alpine 13
.2.1_git20231014'
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.2.1 20231014 (Alpine 13.2.1_git20231014) 
mrVrAlex commented 2 weeks ago

Confirmed, same issue here. This is coredump stacktrace:

swoole::curl::Multi::create_socket (this=this@entry=0x7ffff1400950, cp=0x7ffff147f030, sockfd=sockfd@entry=10) at /tmp/swoole-src/ext-src/swoole_curl.cc:77
77          handle->socket = socket;
(gdb) bt
#0  swoole::curl::Multi::create_socket (this=this@entry=0x7ffff1400950, cp=0x7ffff147f030, sockfd=sockfd@entry=10) at /tmp/swoole-src/ext-src/swoole_curl.cc:77
#1  0x00007ffff5ed677a in swoole::curl::Multi::set_event (this=0x7ffff1400950, cp=0x7ffff147f030, socket_ptr=<optimized out>, sockfd=10, action=1) at /tmp/swoole-src/ext-src/swoole_curl.cc:103
#2  0x00007ffff5ed67b5 in swoole::curl::Multi::handle_socket (easy=<optimized out>, s=<optimized out>, action=<optimized out>, userp=<optimized out>, socketp=<optimized out>) at /tmp/swoole-src/ext-src/swoole_curl.cc:51
#3  0x00007ffff761c1f2 in ?? () from /usr/lib/libcurl.so.4
#4  0x00007ffff75ec4dd in ?? () from /usr/lib/libcurl.so.4
#5  0x00007ffff75ece74 in ?? () from /usr/lib/libcurl.so.4
#6  0x00007ffff75ecf95 in ?? () from /usr/lib/libcurl.so.4
#7  0x00007ffff75ed1bf in ?? () from /usr/lib/libcurl.so.4
#8  0x00007ffff761d0d3 in curl_multi_cleanup () from /usr/lib/libcurl.so.4
#9  0x00007ffff5fafd18 in swoole::curl::Multi::~Multi (this=0x7ffff1400950, __in_chrg=<optimized out>) at /tmp/swoole-src/ext-src/php_swoole_curl.h:112
#10 zif_swoole_native_curl_exec (execute_data=0x7ffff084a480, return_value=0x7ffff1400a00) at /tmp/swoole-src/thirdparty/php/curl/interface.cc:2591
#11 0x00005555559d19bf in execute_ex ()
#12 0x00007ffff5ecf73d in swoole::PHPCoroutine::main_func (arg=0x7fffffff9fa0) at /tmp/swoole-src/ext-src/swoole_coroutine.cc:772
#13 0x00007ffff5f384d4 in std::function<void (void*)>::operator()(void*) const (__args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/13.2.1/bits/std_function.h:591
#14 swoole::coroutine::Context::context_func_v2 (transfer=...) at /tmp/swoole-src/src/coroutine/context.cc:163
#15 0x00005555558f78af in make_fcontext ()
#16 0x0000000000000000 in ?? ()
hydrapolic commented 4 days ago

After migrating to swoole https://github.com/swoole/swoole-src/releases/tag/v5.1.5 it seems to be gone.