search

apache / trafficserver

Apache Traffic Server™ is a fast, scalable and extensible HTTP/1.1 and HTTP/2 compliant caching proxy server.
https://trafficserver.apache.org/
Apache License 2.0
1.82k stars 804 forks source link

header_rewrite plugin crashs on Fedora 27 #3445

Closed thelounge-zz closed 5 years ago

thelounge-zz commented 6 years ago

the same config on Fedora 26 is fine

glibc-2.26-27.fc27.x86_64 versus glibc-2.25-13.fc26.x86_64

[root@testserver:~]$ cat /etc/trafficserver/plugin.config header_rewrite.so /etc/trafficserver/header_rewrite.config

[root@testserver:~]$ cat /etc/trafficserver/header_rewrite.config cond %{SEND_REQUEST_HDR_HOOK} cond %{INCOMING-PORT}=443 set-header X-TL-TLS-Offload 1 set-header X-TL-TLS-Offload-Port %{INCOMING-PORT}

cond %{SEND_REQUEST_HDR_HOOK} cond %{INCOMING-PORT}=8443 set-header X-TL-TLS-Offload 1 set-header X-TL-TLS-Offload-Port %{INCOMING-PORT}

Apr 11 12:45:26 testserver systemd[1]: Stopping Apache Traffic Server... Apr 11 12:45:26 testserver systemd[1]: Stopped Apache Traffic Server. Apr 11 12:45:26 testserver systemd[1]: Starting Apache Traffic Server... Apr 11 12:45:26 testserver systemd[1]: Started Apache Traffic Server. Apr 11 12:45:26 testserver traffic_cop[85223]: --- Cop Starting [Version: Apache Traffic Server - traffic_cop - 7.1.2 - (build # 041112 on Apr 11 2018 at 12:14:02)] --- Apr 11 12:45:26 testserver traffic_cop[85223]: traffic_manager not running, making sure traffic_server is dead Apr 11 12:45:26 testserver traffic_cop[85223]: spawning traffic_manager Apr 11 12:45:26 testserver traffic_manager[85224]: NOTE: --- Manager Starting --- Apr 11 12:45:26 testserver traffic_manager[85224]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 7.1.2 - (build # 041112 on Apr 11 2018 at 12:14:11) Apr 11 12:45:26 testserver traffic_manager[85224]: NOTE: RLIMIT_NOFILE(7):cur(636962),max(636962) Apr 11 12:45:29 testserver traffic_server[85230]: NOTE: --- traffic_server Starting --- Apr 11 12:45:29 testserver traffic_server[85230]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.2 - (build # 041112 on Apr 11 2018 at 12:13:15) Apr 11 12:45:29 testserver traffic_server[85230]: NOTE: RLIMIT_NOFILE(7):cur(636962),max(636962) Apr 11 12:45:29 testserver systemd[1]: Started Process Core Dump (PID 85242/UID 0). Apr 11 12:45:30 testserver systemd-coredump[85243]: Process 85230 (traffic_server) of user 176 dumped core.

Stack trace of thread 85230:

0 0x00007fe811abb660 raise (libc.so.6)

1 0x00007fe811abcd1f abort (libc.so.6)

2 0x00007fe80d4e92b5 n/a (header_rewrite.so)

3 0x00007fe80d4f23f8 _ZN11RulesConfig12parse_configERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID (header_rewrite.so)

4 0x00007fe80d4f289b TSPluginInit (header_rewrite.so)

5 0x0000556cbd9417ba _Z11plugin_initb (traffic_server)

6 0x0000556cbd7d244a main (traffic_server)

7 0x00007fe811aa7f2a __libc_start_main (libc.so.6)

8 0x0000556cbd7d60ca _start (traffic_server)

traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x5643923508a7] /usr/lib64/libpthread.so.0(+0x121b0)[0x7fa2116e11b0] /usr/lib64/libc.so.6(gsignal+0x110)[0x7fa21134d660] /usr/lib64/libc.so.6(abort+0x151)[0x7fa21134ec41] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc2b5)[0x7fa20c9ea2b5] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN11RulesConfig12parse_configERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID+0x1e2)[0x7fa20c9f33f8] /usr/lib64/trafficserver/plugins/header_rewrite.so(TSPluginInit+0xd5)[0x7fa20c9f389b] /usr/bin/traffic_server(_Z11plugin_initb+0x6e3)[0x5643924bb7ba] /usr/bin/traffic_server(main+0x1c2f)[0x56439234c44a] /usr/lib64/libc.so.6(__libc_start_main+0xea)[0x7fa211339f2a] /usr/bin/traffic_server(_start+0x2a)[0x5643923500ca] traffic_server: received signal 6 (Aborted) traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x561cc11818a7] /usr/lib64/libpthread.so.0(+0x121b0)[0x7fedc306e1b0] /usr/lib64/libc.so.6(gsignal+0x110)[0x7fedc2cda660] /usr/lib64/libc.so.6(abort+0x151)[0x7fedc2cdbc41] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc2b5)[0x7fedbe6e92b5] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN11RulesConfig12parse_configERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID+0x1e2)[0x7fedbe6f23f8] /usr/lib64/trafficserver/plugins/header_rewrite.so(TSPluginInit+0xd5)[0x7fedbe6f289b] /usr/bin/traffic_server(_Z11plugin_initb+0x6e3)[0x561cc12ec7ba] /usr/bin/traffic_server(main+0x1c2f)[0x561cc117d44a] /usr/lib64/libc.so.6(libc_start_main+0xea)[0x7fedc2cc6f2a] /usr/bin/traffic_server(_start+0x2a)[0x561cc11810ca] traffic_server: received signal 6 (Aborted) traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x5578c97ff8a7] /usr/lib64/libpthread.so.0(+0x121b0)[0x7f05f1a261b0] /usr/lib64/libc.so.6(gsignal+0x110)[0x7f05f1692660] /usr/lib64/libc.so.6(abort+0x151)[0x7f05f1693c41] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc2b5)[0x7f05ed0e92b5] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN11RulesConfig12parse_configERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID+0x1e2)[0x7f05ed0f23f8] /usr/lib64/trafficserver/plugins/header_rewrite.so(TSPluginInit+0xd5)[0x7f05ed0f289b] /usr/bin/traffic_server(_Z11plugin_initb+0x6e3)[0x5578c996a7ba] /usr/bin/traffic_server(main+0x1c2f)[0x5578c97fb44a] /usr/lib64/libc.so.6(libc_start_main+0xea)[0x7f05f167ef2a] /usr/bin/traffic_server(_start+0x2a)[0x5578c97ff0ca] traffic_server: received signal 6 (Aborted) traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x55c20c3508a7] /usr/lib64/libpthread.so.0(+0x121b0)[0x7faf17a551b0] /usr/lib64/libc.so.6(gsignal+0x110)[0x7faf176c1660] /usr/lib64/libc.so.6(abort+0x151)[0x7faf176c2c41] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc2b5)[0x7faf12dea2b5] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN11RulesConfig12parse_configERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID+0x1e2)[0x7faf12df33f8] /usr/lib64/trafficserver/plugins/header_rewrite.so(TSPluginInit+0xd5)[0x7faf12df389b] /usr/bin/traffic_server(_Z11plugin_initb+0x6e3)[0x55c20c4bb7ba] /usr/bin/traffic_server(main+0x1c2f)[0x55c20c34c44a] /usr/lib64/libc.so.6(__libc_start_main+0xea)[0x7faf176adf2a] /usr/bin/traffic_server(_start+0x2a)[0x55c20c3500ca] traffic_server: received signal 6 (Aborted) traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x557e0c4108a7] /usr/lib64/libpthread.so.0(+0x121b0)[0x7f661b9181b0] /usr/lib64/libc.so.6(gsignal+0x110)[0x7f661b584660] /usr/lib64/libc.so.6(abort+0x151)[0x7f661b585c41] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc2b5)[0x7f6616fea2b5] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN11RulesConfig12parse_configERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE12TSHttpHookID+0x1e2)[0x7f6616ff33f8] /usr/lib64/trafficserver/plugins/header_rewrite.so(TSPluginInit+0xd5)[0x7f6616ff389b] /usr/bin/traffic_server(_Z11plugin_initb+0x6e3)[0x557e0c57b7ba] /usr/bin/traffic_server(main+0x1c2f)[0x557e0c40c44a] /usr/lib64/libc.so.6(__libc_start_main+0xea)[0x7f661b570f2a] /usr/bin/traffic_server(_start+0x2a)[0x557e0c4100ca] [TrafficManager] ==> Cleaning up and reissuing signal #

thelounge-zz commented 6 years ago

since a ton of other services and software (apr, apr-util, mariadb, httpd, mod_security) did run fine with -D_GLIBCXX_ASSERTIONS i obviously copied that to the trafficserver.spec to have it in the next build and did not try the resulting binary until today - this seems to be a security relevant bug in the header_rewrite plugin

gcc-7.3.1-5.fc27.x86_64, same result on Fedora 26 with gcc-7.3.1-2.fc26.x86_64

removing -D_GLIBCXX_ASSERTIONS from the flags below (at bottom the fully translated stuff including %{optflags} which are the global ones on that build-environment) and the crashes are gone

LTO_FLAGS="-flto -ffat-lto-objects -fuse-ld=gold -fuse-linker-plugin" export CFLAGS="%{optflags} -Os -fno-strict-aliasing -ftree-vectorize -ftree-loop-vectorize -ftree-slp-vectorize -falign-functions -falign-jumps -falign-loops -falign-labels -D_GLIBCXX_ASSERTIONS --param=ssp-buffer-size=8 -fPIC $LTO_FLAGS -Wa,--noexecstack -Wno-stack-protector" export CXXFLAGS="$CFLAGS" export CPPFLAGS="$CFLAGS" export CC="gcc $CFLAGS" export CPPFLAGS="-D_FORTIFY_SOURCE=2" export SH_LDFLAGS="-Wl,--as-needed -Wl,-z,now -Wl,-z,relro -Wl,-z,noexecstack %{optflags} -Os -fno-strict-aliasing -ftree-vectorize -ftree-loop-vectorize -ftree-slp-vectorize -falign-functions -falign-jumps -falign-loops -falign-labels -D_GLIBCXX_ASSERTIONS --param=ssp-buffer-size=8 $LTO_FLAGS -Wa,--noexecstack -Wno-stack-protector" export LDFLAGS="$SH_LDFLAGS -pie -fPIE" export GLIBC_TUNABLES="glibc.elision.enable=1"

thelounge-zz commented 6 years ago

the pugin is sometimes unstable even when you build with out -D_GLIBCXX_ASSERTIONS but nobody gives a damn about bugreports here - 7.1.3 crash yesterday, clear software bug in the plugin

Apr 28 22:06:22 proxy kernel: [ET_NET 5][1529]: segfault at 0 ip 00007fa02b9f2398 sp 00007fa02cd0a3e8 error 4 in header_rewrite.so[7fa02b9df000+1f000]

traffic_server: received signal 11 (Segmentation fault) traffic_server - STACK TRACE: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x558e7ea759a3] /usr/lib64/libpthread.so.0(+0x12420)[0x7fa031006420] /usr/lib64/trafficserver/plugins/header_rewrite.so(_Z7getPortPK8sockaddr+0x0)[0x7fa02b9f2398] /usr/lib64/trafficserver/plugins/header_rewrite.so(_ZN21ConditionIncomingPort4evalERK9Resources+0x1d)[0x7fa02b9f24b3] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0xc652)[0x7fa02b9eb652] /usr/lib64/trafficserver/plugins/header_rewrite.so(+0x11520)[0x7fa02b9f0520] /usr/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xbb)[0x558e7ea75bf1] /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x41e)[0x558e7eb913f4] /usr/bin/traffic_server(_ZN6HttpSM23handle_http_server_openEv+0x1a1)[0x558e7eb5192d] /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x383)[0x558e7eb8ca7f] /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x143)[0x558e7eb8be3f] /usr/bin/traffic_server(_ZN18UnixNetVConnection9connectUpEP7EThreadi+0x28b)[0x558e7eabb1a7] /usr/bin/traffic_server(_ZN16UnixNetProcessor19connect_re_internalEP12ContinuationPK8sockaddrP12NetVCOptions+0x397)[0x558e7eabf7b7] /usr/bin/traffic_server(_ZN6HttpSM19do_http_server_openEb+0x1265)[0x558e7eb52bb5] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xbe7)[0x558e7eb53915] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM22state_cache_open_writeEiPv+0x3db)[0x558e7eb8c263] /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x143)[0x558e7eb8be3f] /usr/bin/traffic_server(_ZN11HttpCacheSM22state_cache_open_writeEiPv+0x1de)[0x558e7eb9f1c8] /usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x43)[0x558e7eaca379] /usr/bin/traffic_server(_ZN7CacheVC18openWriteStartDoneEiP5Event+0x3b7)[0x558e7eab069f] /usr/bin/traffic_server(_ZN5Cache10open_writeEP12ContinuationPKN3ats10CryptoHashEP8HTTPInfolS5_13CacheFragTypePKci+0x36f)[0x558e7eab028f] /usr/bin/traffic_server(_ZN11HttpCacheSM10open_writeEPK12HttpCacheKeyP3URLP7HTTPHdrP8HTTPInfolbb+0xdb)[0x558e7eb9ee6f] /usr/bin/traffic_server(_ZN6HttpSM23do_cache_prepare_actionEP11HttpCacheSMP8HTTPInfobb+0x1b7)[0x558e7eb4d7d3] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe12)[0x558e7eb53b40] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e4)[0x558e7eb92bc4] /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x5a5)[0x558e7eb9157b] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xa18)[0x558e7eb53746] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x366)[0x558e7eb4e746] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb6e)[0x558e7eb5389c] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e4)[0x558e7eb92bc4] /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x5a5)[0x558e7eb9157b] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xa18)[0x558e7eb53746] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e4)[0x558e7eb92bc4] /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x5a5)[0x558e7eb9157b] /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xa18)[0x558e7eb53746] /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x114)[0x558e7eb4969e] /usr/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x1e7)[0x558e7eb8d4b3] /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x143)[0x558e7eb8be3f] /usr/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x191)[0x558e7eb9f451] /usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x43)[0x558e7eaca379] /usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0x676)[0x558e7eb1e854] /usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0x62b)[0x558e7eb1e809] /usr/bin/traffic_server(_ZN7CacheVC17openReadStartHeadEiP5Event+0x941)[0x558e7eb1f1d3] /usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x976)[0x558e7eaa4a9e] /usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x1f)[0x558e7eac1b3b] /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x63)[0x558e7ebcd2fb] /usr/bin/traffic_server(_ZN7EThread13process_queueEP5QueueI5EventNS1_9Link_linkEE+0x53)[0x558e7ebcd3e7] /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x4e)[0x558e7ebcd51a] /usr/bin/traffic_server(+0x1fbf78)[0x558e7ebcbf78] /usr/lib64/libpthread.so.0(+0x736d)[0x7fa030ffb36d] /usr/lib64/libc.so.6(clone+0x3f)[0x7fa030d2fb4f]

zwoop commented 5 years ago

Is this still an issue?

thelounge-zz commented 5 years ago

seems to work with 8.0.1 so far which is the reason i stay at this version even with the somehow broken https-keep-alive while 8.0.2 is completly unuseable (https://github.com/apache/trafficserver/issues/4921#issuecomment-462995734)

bryancall commented 5 years ago

Looks like this has been fixed. Please reopen if there is still an issue.