Haproxy 2.8.3 consuming more than 100% CPU

mvgadagi commented 10 months ago

Detailed Description of the Problem

Haproxy consumes more than 140% of CPU for a very low volume of connections.

I observed this from the htop/top command in the Linux

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5397 root 20 0 1046340 114544 3760 S 147.1 0.2 1:45.03 haproxy

Note : I have around 2000 backends

Haproxy Stats connections output

pid = 80 (process #1, nbproc = 1, nbthread = 14) uptime = 0d 0h07m44s; warnings = 265 system limits: memmax = unlimited; ulimit-n = 19544 maxsock = 19544; maxconn = 8000; reached = 0; maxpipes = 0 current conns = 276; current pipes = 0/0; conn rate = 69/sec; bit rate = 185.750 Mbps Running tasks: 0/6385; idle = 89 %

Expected Behavior

Haproxy shouldn't consume more memory, as the number of connections (max 800 to 1000) is much less in my case

Steps to Reproduce the Behavior

Build the Image and run steps

# build stage
FROM dockerhub.myprivateregistry/debian:buster-slim as build-stage

ENV SOURCE_DIR=/usr/src

RUN set -x \
    \
    && apt-get update \
    && apt-get -y install \
        wget \
        unzip \
        make \
        gcc \
        liblua5.3-dev \
        libssl-dev \
        \
    && rm -rf /var/lib/apt/lists/* \
    \
    && echo "[*] Downloading lua dependency wahern-luaossl" \
    && cd ${SOURCE_DIR} \
    && wget https://github.com/wahern/luaossl/archive/master.zip \
    && unzip -o master.zip \
    && rm -f master.zip \
    && echo "[*] Building and installing lua dependency wahern-luaossl" \
    && cd ${SOURCE_DIR}/luaossl-master \
    && make install \
    \
    && echo "[*] Downloading lua dependency diegonehab-luasocket" \
    && cd ${SOURCE_DIR} \
    && wget https://github.com/diegonehab/luasocket/archive/master.zip \
    && unzip -o master.zip \
    && rm -f master.zip \
    && echo "[*] Building and installing lua dependency diegonehab-luasocket" \
    && cd ${SOURCE_DIR}/luasocket-master \
    && make linux install LUAV=5.3 \
    \
    && echo "[*] Cleanup" \
    && cd ${SOURCE_DIR} \
    && rm -rf ${SOURCE_DIR}/* \
    && cd /

# production stage
FROM dockerhub.myprivateregistry/registry-1-docker-io-remote/haproxy:2.8.3
USER root
RUN usermod -aG root haproxy
RUN mkdir -p /run
RUN chown -R haproxy:haproxy /run
RUN mkdir -p /var/lib/haproxy
COPY --from=build-stage /usr/local/lib/lua/5.3 /usr/local/lib/lua/5.3
COPY --from=build-stage /usr/local/share/lua/5.3 /usr/local/share/lua/5.3
COPY lua /usr/local/share/lua/5.3/

EXPOSE 80 443 9090
CMD [ "-W", "-f", "/etc/haproxy/haproxy.cfg" ]

Do you have any idea what may have caused this?

I don't have any clue, I have gone through the community support and I followed all the suggestions but still the same issueI

Do you have an idea how to solve the issue?

No response

What is your configuration?

#---------------------------------------------------------------------
Global settings
#---------------------------------------------------------------------

global
log /dev/log local1 info alert

pidfile /run/haproxy.pid
maxconn 8000
lua-load /usr/local/share/lua/5.3/accessControl.lua

turn on stats unix socket
stats socket /var/lib/haproxy/stats
stats socket 127.0.0.1:14567

utilize system-wide crypto-policies
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:EECDH+AESGCM:EECDH+CHACHA20
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
tune.bufsize 32768

setenv ISSUER https://auth.myapp.com/auth/realms/rtr
setenv AUDIENCE account
setenv PUBKEY /etc/haproxy/pem/rtr.pem
setenv TOKEN_NAME token-rtr

#---------------------------------------------------------------------
common defaults that all the ‘listen’ and ‘backend’ sections will
use if not designated in their block
#---------------------------------------------------------------------

defaults
option forwardfor except 127.0.0.1
mode http
log global
option httplog
retries 3
option http-server-close
timeout http-request 10s
timeout queue 10m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout http-keep-alive 10s
timeout check 10s
log-format %[var(txn.identifier)]\ %ci:%cp\ %fi:%fp\ %b\ %si:%sp\ %ST\ %HM\ %HU
balance roundrobin

never fail on address resolution
default-server init-addr last,libc,none

#---------------------------------------------------------------------
stats page
#---------------------------------------------------------------------

listen stats
bind :9090
mode http
stats enable # Enable stats page
stats hide-version # Hide HAProxy version
stats uri /
stats auth myuser:mypassword

#---------------------------------------------------------------------
main frontenda which proxys to the backends
#---------------------------------------------------------------------

frontend http-frontend
bind :80
redirect scheme https if !{ ssl_fc } !{ hdr(host) test .myapp.com }
use_backend test if { hdr(host) test.myapp.com }

frontend https-frontend
bind :443 ssl crt /etc/haproxy/certs/

bind :443 ssl crt /etc/haproxy/ssl/haproxy_ssl.pem
#---------------------------------------------------------------------
Default list of internal micro service backend
#---------------------------------------------------------------------

use_backend sirius if { hdr(host),lower,word(1,:slight_smile: sirius.myapp.com }

backend withlogin
http-request lua.simpleAuth
acl cookie var(txn.authorized) -m bool
acl apikey req.hdr(MY-Key) -m str -f /etc/haproxy/keys/cb-exporter
http-request set-var(txn.identifier) str(cb-exporter) if apikey
http-request redirect code 302 location https://login.myapp.com?next_url=%[hdr(host)]%[capture.req.uri,regsub(&,%26,g)] unless cookie || apikey
cookie SERVERID insert indirect nocache
server withlogin0 myhost.com:18091 ssl verify none check cookie withlogin0
backend myapp
server myapp0 myhost.com:443 ssl verify none check

Output of `haproxy -vv`

HAProxy version 2.8.3-86e043a 2023/09/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.3.html
Running on: Linux 3.10.0-1160.88.1.el7.x86_64 #1 SMP Sat Feb 18 13:27:00 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=14).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

Server Configuration

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 14 On-line CPU(s) list: 0-13 Thread(s) per core: 1 Core(s) per socket: 7 Socket(s): 2 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz Stepping: 1 CPU MHz: 2596.992 BogoMIPS: 5193.98 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 35840K NUMA node0 CPU(s): 0-13

wtarreau commented 10 months ago

I'm not seeing anything suspicious in your config. Is this CPU usage something new ? If it is consecutive to an upgrade, what was the last version not showing it ? Does it seem to follow the traffic, or to evolve over time ? Otherwise, could you please share the output of "show info", "show activity" and "show fd" on the CLI ? I'd like to spot if something seems stuck for example. The output of a "perf top" could probably help as well.

lukastribus commented 10 months ago

2.2.17 was used originally, then 2.2.31 and subsequently 2.8.3.

A LUA script using forbidden filesystems functions for every requests was change to use hard coded values:

https://discourse.haproxy.org/t/haproxy-always-consumes-99-100-cpu/9013/12

mvgadagi commented 10 months ago

Thank you @lukastribus for joining here us as well.

@wtarreau @lukastribus Just to add I don't think it's due to an increase in the traffic. We have 2 haproxy nodes and we have observed this in node 1 even when the whole traffic was passed to node 2.

Please have a look at the details below

show info 
Name: HAProxy
Version: 2.8.3-86e043a
Release_date: 2023/09/07
Nbthread: 14
Nbproc: 1
Process_num: 1
Pid: 13
Uptime: 0d 0h02m26s
Uptime_sec: 146
Memmax_MB: 0
PoolAlloc_MB: 1
PoolUsed_MB: 1
PoolFailed: 0
Ulimit-n: 19540
Maxsock: 19540
Maxconn: 8000
Hard_maxconn: 8000
CurrConns: 0
CumConns: 239212
CumReq: 5
MaxSslConns: 0
CurrSslConns: 163
CumSslConns: 209459
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 1
SessRate: 0
SessRateLimit: 0
MaxSessRate: 1
SslRate: 0
SslRateLimit: 0
MaxSslRate: 1
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 1
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 798
SslBackendMaxKeyRate: 1490
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 5894
Run_queue: 1
Idle_pct: 94
node: haproxy
Stopping: 0
Jobs: 223
Unstoppable Jobs: 1
Listeners: 7
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 120947113
TotalSplicedBytesOut: 0
BytesOutRate: 708288
DebugCommandsIssued: 0
CumRecvLogs: 0
Build info: 2.8.3-86e043a
Memmax_bytes: 0
PoolAlloc_bytes: 2019120
PoolUsed_bytes: 2019120
Start_time_sec: 1694956790
Tainted: 0x40
TotalWarnings: 251
MaxconnReached: 0
BootTime_ms: 20241

show pools
Dumping pools usage. Use SIGQUIT to flush them.
  - Pool hlua_queuew (24 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 7 users, @0x55d04618db00 [SHARED]
  - Pool hlua_flt_ct (56 bytes) : 3014 allocated (168784 bytes), 3014 used (~2274 by thread caches), needed_avg 3009, 0 failures, 17 users, @0x55d046189a40 [SHARED]
  - Pool hlua_esub (88 bytes) : 436 allocated (38368 bytes), 436 used (~330 by thread caches), needed_avg 432, 0 failures, 9 users, @0x55d046181880 [SHARED]
  - Pool ssl_sock_ct (120 bytes) : 382 allocated (45840 bytes), 382 used (~294 by thread caches), needed_avg 378, 0 failures, 4 users, @0x55d04617d7c0 [SHARED]
  - Pool hlua (152 bytes) : 6767 allocated (1028584 bytes), 6767 used (~663 by thread caches), needed_avg 6763, 0 failures, 5 users, @0x55d046185980 [SHARED]
  - Pool h2s (184 bytes) : 439 allocated (80776 bytes), 439 used (~334 by thread caches), needed_avg 435, 0 failures, 5 users, @0x55d046191bc0 [SHARED]
  - Pool tcpcheck_ru (216 bytes) : 4 allocated (864 bytes), 4 used (~4 by thread caches), needed_avg 1, 0 failures, 4 users, @0x55d04619de40 [SHARED]
  - Pool ssl_sock_cl (280 bytes) : 3 allocated (840 bytes), 3 used (~2 by thread caches), needed_avg 1, 0 failures, 4 users, @0x55d046179700 [SHARED]
  - Pool resolv_reso (472 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d0461a6000 [SHARED]
  - Pool resolv_answ (600 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d0461aa0c0 [SHARED]
  - Pool ehdl_sub_ed (792 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d0461b2240 [SHARED]
  - Pool stream (888 bytes) : 4 allocated (3552 bytes), 4 used (~3 by thread caches), needed_avg 1, 0 failures, 1 users, @0x55d0461a1f00 [SHARED]
  - Pool requri (1048 bytes) : 4 allocated (4192 bytes), 4 used (~4 by thread caches), needed_avg 1, 0 failures, 1 users, @0x55d0486e5680 [SHARED]
  - Pool fcgi_conn (1240 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d046199d80 [SHARED]
  - Pool h2c (1304 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d046195cc0 [SHARED]
  - Pool hpack_tbl (4096 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d0498f3780 [SHARED]
  - Pool buffer (32768 bytes) : 8 allocated (262144 bytes), 8 used (~6 by thread caches), needed_avg 4, 0 failures, 1 users, @0x55d0498ef6c0 [SHARED]
  - Pool trash (32800 bytes) : 14 allocated (459200 bytes), 14 used (~13 by thread caches), needed_avg 10, 0 failures, 1 users, @0x55d0461b6340
  - Pool dns_msg_buf (65560 bytes) : 0 allocated (0 bytes), 0 used (~0 by thread caches), needed_avg 0, 0 failures, 1 users, @0x55d0461ae180 [SHARED]
Total: 19 pools, 2093144 bytes allocated, 2093144 used (~985184 by thread caches).

show activity
thread_id: 4 (1..14)
date_now: 1694957053.285216
uptime_now: 242.517594
ctxsw: 2276046 [ 161355 160709 165026 167903 160082 164004 166150 161460 161903 164112 164815 158773 158314 161440 ]
tasksw: 820264 [ 57716 58928 59047 59473 57018 58972 60536 57750 58461 59159 60094 56901 57267 58942 ]
empty_rq: 32237 [ 2444 2280 2338 2252 2329 2260 2228 2354 2394 2250 2154 2312 2292 2350 ]
long_rq: 734102 [ 52209 51542 53206 54569 51834 52727 53124 52264 52297 53131 53143 51301 50979 51776 ]
loops: 1790803 [ 127150 126495 129876 132146 126041 128920 130641 127293 127391 128916 129178 125188 124554 127014 ]
wake_tasks: 734103 [ 52209 51542 53206 54569 51834 52727 53124 52265 52297 53131 53143 51301 50979 51776 ]
wake_signal: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
poll_io: 884305 [ 63032 61996 64122 65320 62005 64160 64970 63173 62866 63637 63442 61572 61390 62620 ]
poll_exp: 20 [ 0 2 1 2 3 1 2 1 1 2 1 2 1 1 ]
poll_drop_fd: 137 [ 8 5 9 12 8 12 14 9 12 8 7 11 12 10 ]
poll_skip_fd: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
conn_dead: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
stream_calls: 24 [ 6 5 6 4 3 0 0 0 0 0 0 0 0 0 ]
pool_fail: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
buf_wait: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
cpust_ms_tot: 136 [ 21 13 0 4 0 24 6 5 14 8 11 1 13 16 ]
cpust_ms_1s: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
cpust_ms_15s: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
avg_cpu_pct: 5 [ 5 4 6 4 5 7 5 5 6 4 3 3 3 4 ]
avg_loop_us: 135 [ 142 136 134 135 132 137 134 133 145 136 135 132 131 134 ]
accepted: 9 [ 0 0 3 1 2 0 0 0 0 0 0 3 0 0 ]
accq_pushed: 9 [ 2 2 2 2 1 0 0 0 0 0 0 0 0 0 ]
accq_full: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
accq_ring: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]
fd_takeover: 0 [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ]

show fd

     5 : st=0x020023(Cl heopi W:sRa R:sRA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d046213ba0 iocb=0x55d045b00160(mworker_accept_wrapper)
      7 : st=0x120023(Cl heopi W:sRa R:sRA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d046242ad0 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=GLOBAL fam=ipv4 lport=9090
      8 : st=0x120023(Cl heopi W:sRa R:sRA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d046249cd0 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=http-frontend fam=ipv4 lport=80
      9 : st=0x120121(Cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d04624ca10 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=https-frontend fam=ipv4 lport=443
     10 : st=0x120023(Cl heopi W:sRa R:sRA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d0486dc4b0 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=apiv2 fam=ipv4 lport=20002
     11 : st=0x120023(Cl heopi W:sRa R:sRA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d0486df620 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=si fam=ipv4 lport=18033
     12 : st=0x120121(Cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x3fff umask=0x0 prmsk=0x3fff pwmsk=0x0 owner=0x55d046243060 iocb=0x55d045aff560(sock_accept_iocb) l.st=RDY fe=GLOBAL fam=unix
     13 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x2 umask=0x0 prmsk=0x2 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     14 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x2 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     16 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     17 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     19 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     20 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     22 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     23 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     25 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x200 umask=0x0 prmsk=0x200 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     26 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x200 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     28 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     29 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     31 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     32 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     34 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x1000 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     35 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     37 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x40 umask=0x0 prmsk=0x40 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     38 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x40 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     40 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     41 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     42 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     43 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     45 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x80 umask=0x0 prmsk=0x80 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     46 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x80 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     48 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x10 umask=0x0 prmsk=0x10 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     49 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x10 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     51 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x20 umask=0x0 prmsk=0x20 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     52 : st=0x000022(cl heopi W:sRa R:sRa) ref=0 gid=1 tmask=0x20 umask=0x0 prmsk=0x0 pwmsk=0x0 owner=0x55d045b07a20 iocb=0x55d045b07a20(poller_pipe_io_handler)
     54 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x0 owner=0x7f41700dd370 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=51064 rport=31003 sv=grafana.ba-prd-usc1c-c-0.gcp-1a-20190412-01.gce/grafana.ba-prd-usc1c-c-0.gcp-1a-20190412-01.gce0 mux=PASS ctx=0x7f41700580f0 xprt=SSL xprt_ctx=0x7f41700530f0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4170115938(ev=1 tl=0x7f41700f32b0 tl.calls=0 tl.ctx=0x7f4170115920 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     56 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x7f4154052930 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=45362 rport=443 sv=api-splunk-prd-we-msntcytricb/api-splunk-prd-we-msntcytricb0 mux=PASS ctx=0x7f415405ec00 xprt=SSL xprt_ctx=0x7f4154063ea0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4154063ad8(ev=1 tl=0x7f415404db80 tl.calls=0 tl.ctx=0x7f4154063ac0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     57 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x8 owner=0x7f416805bff0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=41222 sv=cur-uat-6x-jaeger-query-xdlc/cur-uat-6x-jaeger-query-xdlc0 mux=PASS ctx=0x7f4168129f30 xprt=SSL xprt_ctx=0x7f4168051f20 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41680c9508(ev=1 tl=0x7f416805e7f0 tl.calls=0 tl.ctx=0x7f41680c94f0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     58 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x1000 pwmsk=0x1000 owner=0x7f415c07be00 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=50272 sv=orchestrator-uat-api/orchestrator-uat-api0 mux=PASS ctx=0x7f415c050230 xprt=SSL xprt_ctx=0x7f415c0d2280 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415c0a0728(ev=1 tl=0x7f415c05d460 tl.calls=0 tl.ctx=0x7f415c0a0710 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     62 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x2000 owner=0x7f41640a3420 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=53924 sv=jclfsandboxms.forge.amadeus.net/jclfsandboxms.forge.amadeus.net0 mux=PASS ctx=0x7f41640db580 xprt=SSL xprt_ctx=0x7f416410eef0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f4164083908(ev=1 tl=0x7f41640d94c0 tl.calls=0 tl.ctx=0x7f41640838f0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     67 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x20 umask=0x0 prmsk=0x20 pwmsk=0x0 owner=0x7f41401fcfe0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=48966 rport=443 sv=monitoring-console-splunk-prd-we-msntcytrica/monitoring-console-splunk-prd-we-msntcytrica0 mux=PASS ctx=0x7f4140053ba0 xprt=SSL xprt_ctx=0x7f4140052b50 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f414015ac28(ev=1 tl=0x7f414004efa0 tl.calls=0 tl.ctx=0x7f414015ac10 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     68 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x0 owner=0x7f415805d060 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=58792 rport=443 sv=master-splunk-rnd-ne-ima02a/master-splunk-rnd-ne-ima02a0 mux=PASS ctx=0x7f4158057480 xprt=SSL xprt_ctx=0x7f41580595b0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f41580aa0b8(ev=1 tl=0x7f4158104640 tl.calls=0 tl.ctx=0x7f41580aa0a0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     69 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x7f4168083fb0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=41090 rport=443 sv=cur-pdt-ay-orms-recovery-tool-xdlc/cur-pdt-ay-orms-recovery-tool-xdlc0 mux=PASS ctx=0x7f4168082620 xprt=SSL xprt_ctx=0x7f416806bbb0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f416805bcb8(ev=1 tl=0x7f41681a9060 tl.calls=0 tl.ctx=0x7f416805bca0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     70 : st=0x000121(cl heopI W:sRa R:srA) ref=0 gid=1 tmask=0x80 umask=0x80 prmsk=0x80 pwmsk=0x0 owner=0x7f415416a2f0 iocb=0x55d045affab0(sock_conn_iocb) back=0 cflg=0x00000300 cerr=0 fam=unix fe=GLOBAL mux=PASS ctx=0x7f414805bcc0 xprt=RAW
     72 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x2000 owner=0x7f4164082480 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=38956 sv=prometheus.ngi-prd-euw3b-c-39.ngi-box-prd.gce/prometheus.ngi-prd-euw3b-c-39.ngi-box-prd.gce0 mux=PASS ctx=0x7f4164113e00 xprt=RAW
     74 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x7f4168097e00 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=54542 rport=32600 sv=boxapi.nh-prd-usc1-r-6.amadeus-nh-sabre-prd.gce/boxapi.nh-prd-usc1-r-6.amadeus-nh-sabre-prd.gce0 mux=PASS ctx=0x7f416806c1f0 xprt=SSL xprt_ctx=0x7f41681acc00 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4168095768(ev=1 tl=0x7f416806cb70 tl.calls=0 tl.ctx=0x7f4168095750 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     75 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x8 owner=0x7f4168061090 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=38374 sv=splunk-tst-we-int08/splunk-tst-we-int080 mux=PASS ctx=0x7f4168152100 xprt=SSL xprt_ctx=0x7f41680aecf0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41681db9e8(ev=1 tl=0x7f4168117b60 tl.calls=0 tl.ctx=0x7f41681db9d0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     76 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x100 owner=0x7f416c061470 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=34682 sv=bird-uat/bird-uat0 mux=PASS ctx=0x7f416c06e860 xprt=RAW
     77 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x0 owner=0x7f4164110760 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=46916 rport=443 sv=watchtower/watchtower0 mux=PASS ctx=0x7f4164084a20 xprt=SSL xprt_ctx=0x7f41640596b0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f41641109b8(ev=1 tl=0x7f416404dca0 tl.calls=0 tl.ctx=0x7f41641109a0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     78 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x1 owner=0x55d049c641e0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=42836 sv=splunk-rnd-ne-ima02b/splunk-rnd-ne-ima02b0 mux=PASS ctx=0x55d049bedbe0 xprt=SSL xprt_ctx=0x55d049c2a9f0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x55d049bd49b8(ev=1 tl=0x55d049c196f0 tl.calls=0 tl.ctx=0x55d049bd49a0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     79 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x7f4168050400 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=54668 rport=18091 sv=sklttr.cb/sklttr.cb0 mux=PASS ctx=0x7f41680dafe0 xprt=SSL xprt_ctx=0x7f41680549a0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f416804f838(ev=1 tl=0x7f4168126d30 tl.calls=0 tl.ctx=0x7f416804f820 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     80 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x100 owner=0x7f416c072400 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=41480 sv=prometheus.ay1s-prd-usc1c-c-4.amadeus-prd-2b385f.gce/prometheus.ay1s-prd-usc1c-c-4.amadeus-prd-2b385f.gce0 mux=PASS ctx=0x7f416c04dcf0 xprt=RAW
     81 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x7f41540ecba0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=47310 rport=443 sv=forge3/forge30 mux=PASS ctx=0x7f4154068990 xprt=SSL xprt_ctx=0x7f415404ddf0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f415410e618(ev=1 tl=0x7f4154061480 tl.calls=0 tl.ctx=0x7f415410e600 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     82 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x80 umask=0x0 prmsk=0x80 pwmsk=0x80 owner=0x7f4148050280 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=32940 sv=custom-seco-seco-bk-bery-acc.dmz.nld8/custom-seco-seco-bk-bery-acc.dmz.nld80 mux=PASS ctx=0x7f41480a7410 xprt=SSL xprt_ctx=0x7f4148058e90 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41480500d8(ev=1 tl=0x7f41480599f0 tl.calls=0 tl.ctx=0x7f41480500c0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     83 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x20 umask=0x0 prmsk=0x20 pwmsk=0x20 owner=0x7f414005db90 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=33484 sv=prometheus.ay1s-prd-usc1b-c-4.amadeus-prd-2b385f.gce/prometheus.ay1s-prd-usc1b-c-4.amadeus-prd-2b385f.gce0 mux=PASS ctx=0x7f4140050020 xprt=RAW
     84 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x1 owner=0x55d049c2ae20 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=53682 sv=prometheus.mh-prd-usc1-r-7.noted-victory-789.gce/prometheus.mh-prd-usc1-r-7.noted-victory-789.gce0 mux=PASS ctx=0x55d049c5a700 xprt=RAW
     85 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x4 owner=0x7f41700c9a70 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=56012 rport=80 sv=topx.muctxpv001/topx.muctxpv0010 mux=PASS ctx=0x7f417005d410 xprt=RAW
     86 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x7f4154108f90 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=37332 rport=18091 sv=devasc.cb/devasc.cb1 mux=PASS ctx=0x7f4154099d00 xprt=SSL xprt_ctx=0x7f4154068c00 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f415418b6d8(ev=1 tl=0x7f415405c660 tl.calls=0 tl.ctx=0x7f415418b6c0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     87 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x1000 pwmsk=0x1000 owner=0x7f415c087050 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=45624 sv=grafana.ct-prd-ase1b-c-11.gcp-1a-20201028-01.gce/grafana.ct-prd-ase1b-c-11.gcp-1a-20201028-01.gce0 mux=PASS ctx=0x7f415c0d6f20 xprt=SSL xprt_ctx=0x7f415c0a0980 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415c074498(ev=1 tl=0x7f415c07d840 tl.calls=0 tl.ctx=0x7f415c074480 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     88 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x7f416804ef80 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=51572 rport=443 sv=prd-polaris-orms-recovery-tool-xdlc/prd-polaris-orms-recovery-tool-xdlc0 mux=PASS ctx=0x7f416806bc90 xprt=SSL xprt_ctx=0x7f4168159000 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f416804dc58(ev=1 tl=0x7f416806ccf0 tl.calls=0 tl.ctx=0x7f416804dc40 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     90 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x40 umask=0x0 prmsk=0x40 pwmsk=0x40 owner=0x7f4150073080 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39648 sv=uniping-watchdog-fra/uniping-watchdog-fra0 mux=PASS ctx=0x7f4150107a20 xprt=SSL xprt_ctx=0x7f41500777a0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41500bf898(ev=1 tl=0x7f4150186880 tl.calls=0 tl.ctx=0x7f41500bf880 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     91 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x0 owner=0x7f416804dac0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=37088 rport=443 sv=splunk-prd-wus2-ahcrs01a/splunk-prd-wus2-ahcrs01a0 mux=PASS ctx=0x55d049bef8d0 xprt=SSL xprt_ctx=0x55d049c1be30 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x55d049becd28(ev=1 tl=0x55d049c46ed0 tl.calls=0 tl.ctx=0x55d049becd10 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     92 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x80 umask=0x0 prmsk=0x80 pwmsk=0x0 owner=0x7f414812d9f0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x00000302 cerr=0 fam=ipv4 lport=49382 rport=443 sv=nceratt/nceratt0 mux=H1 ctx=0x7f4148069e10 h1c.flg=0x80008800 .sub=1 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 .task=0x7f4148066d60 .exp=<NEVER> xprt=SSL xprt_ctx=0x7f41480504a0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4148069eb0(ev=1 tl=0x7f414804db30 tl.calls=3 tl.ctx=0x7f4148069e10 tl.fct=h1_io_cb) .sent_early=0 .early_in=0
     93 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x0 owner=0x7f4158052930 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=36886 rport=32600 sv=boxapi.ct-prd-ase1b-c-11.gcp-1a-20201028-01.gce/boxapi.ct-prd-ase1b-c-11.gcp-1a-20201028-01.gce0 mux=PASS ctx=0x7f4158050070 xprt=SSL xprt_ctx=0x7f4158055200 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4158052788(ev=1 tl=0x7f41580527e0 tl.calls=0 tl.ctx=0x7f4158052770 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     94 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x0 owner=0x7f416c0619a0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=45590 rport=443 sv=hec-splunk-prd-eus2-hosobs01a/hec-splunk-prd-eus2-hosobs01a0 mux=PASS ctx=0x7f416c05b0d0 xprt=SSL xprt_ctx=0x7f416c061a70 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f416c088118(ev=1 tl=0x7f416c137830 tl.calls=0 tl.ctx=0x7f416c088100 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     95 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x400 owner=0x7f4158107540 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=35730 sv=restktpaya-dev-clf-microservice.app.tst-ne-restktpaya.restktpay.northeurope.tstcur.tnz.amadeus.net/restktpaya-dev-clf-microservice.app.tst-ne-restktpaya.restktpay.northeurope.tstcur.tnz.amadeus.net0 mux=PASS ctx=0x7f4158077db0 xprt=SSL xprt_ctx=0x7f41580dce80 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f4158108e48(ev=1 tl=0x7f415810e5a0 tl.calls=0 tl.ctx=0x7f4158108e30 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     96 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x1000 pwmsk=0x1000 owner=0x7f415c054fe0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=49936 sv=grafana.ct-prd-ase1a-c-10.gcp-1a-20201028-01.gce/grafana.ct-prd-ase1a-c-10.gcp-1a-20201028-01.gce0 mux=PASS ctx=0x7f415c04e6e0 xprt=SSL xprt_ctx=0x7f415c07e550 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415c054e38(ev=1 tl=0x7f415c056920 tl.calls=0 tl.ctx=0x7f415c054e20 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     98 : st=0x010121(cL heopI W:sRa R:srA) ref=0 gid=1 tmask=0x80 umask=0x0 prmsk=0x80 pwmsk=0x0 owner=0x7f414806a010 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=46112 rport=443 sv=dga-security-demo/dga-security-demo0 mux=PASS ctx=0x7f414805c610 xprt=SSL xprt_ctx=0x7f41480d89c0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f414812dad8(ev=1 tl=0x7f41480bf720 tl.calls=0 tl.ctx=0x7f414812dac0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
     99 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x100 owner=0x7f416c071fb0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=42842 sv=master-splunk-rnd-ne-ima02b/master-splunk-rnd-ne-ima02b0 mux=PASS ctx=0x7f416c05f1f0 xprt=SSL xprt_ctx=0x7f416c070d30 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f416c05c9f8(ev=1 tl=0x7f416c0c83e0 tl.calls=0 tl.ctx=0x7f416c05c9e0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    101 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x0 owner=0x7f416805ac30 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=40962 rport=443 sv=cur-dev-aa-orms-kbrowse-xdlc/cur-dev-aa-orms-kbrowse-xdlc0 mux=PASS ctx=0x7f416815b6b0 xprt=SSL xprt_ctx=0x7f4168097c00 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f416806a0f8(ev=1 tl=0x7f416806a410 tl.calls=0 tl.ctx=0x7f416806a0e0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    102 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x400 owner=0x7f415807c970 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=56560 sv=lbe-sip-tst-admin/lbe-sip-tst-admin0 mux=PASS ctx=0x7f4158065f80 xprt=RAW
    103 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x100 umask=0x0 prmsk=0x100 pwmsk=0x100 owner=0x7f416c0f8390 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=41142 sv=cur-dev-iso-mongo-express-xdlc/cur-dev-iso-mongo-express-xdlc0 mux=PASS ctx=0x7f416c0c0b10 xprt=SSL xprt_ctx=0x7f416c088210 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f416c0f8678(ev=1 tl=0x7f416c05efd0 tl.calls=0 tl.ctx=0x7f416c0f8660 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    104 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x200 umask=0x0 prmsk=0x200 pwmsk=0x0 owner=0x7f41600aeb90 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=41110 rport=443 sv=cur-pdt-ay-skube-injector-xdlc/cur-pdt-ay-skube-injector-xdlc0 mux=PASS ctx=0x7f41600887b0 xprt=SSL xprt_ctx=0x7f4160084530 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f41600e5138(ev=1 tl=0x7f41600ce4f0 tl.calls=0 tl.ctx=0x7f41600e5120 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    107 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x20 umask=0x0 prmsk=0x20 pwmsk=0x20 owner=0x7f41400fa160 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=59796 rport=443 sv=splunk-prd-we-paym01b/splunk-prd-we-paym01b0 mux=PASS ctx=0x7f41400714d0 xprt=SSL xprt_ctx=0x7f414006dc50 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41400fc508(ev=1 tl=0x7f414009f040 tl.calls=0 tl.ctx=0x7f41400fc4f0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    109 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x7f4154082f80 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=47216 rport=30157 sv=testrgrid/testrgrid1 mux=PASS ctx=0x7f415405c1e0 xprt=SSL xprt_ctx=0x7f415406b2b0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f415405c248(ev=1 tl=0x7f4154085380 tl.calls=0 tl.ctx=0x7f415405c230 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    110 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x10 umask=0x0 prmsk=0x10 pwmsk=0x10 owner=0x7f414c071c90 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=59756 sv=splunk-tst-ne-icna01b/splunk-tst-ne-icna01b0 mux=PASS ctx=0x7f414c192e90 xprt=SSL xprt_ctx=0x7f414c0f7200 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f414c053a98(ev=1 tl=0x7f414c0d3510 tl.calls=0 tl.ctx=0x7f414c053a80 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    114 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x0 owner=0x7f417010a8a0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=45060 rport=32600 sv=boxapi.ct-prd-ase1a-c-b.gcp-1a-20201028-01.gce/boxapi.ct-prd-ase1a-c-b.gcp-1a-20201028-01.gce0 mux=PASS ctx=0x7f4170079fe0 xprt=SSL xprt_ctx=0x7f41700eceb0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f417005cd88(ev=1 tl=0x7f417004f3f0 tl.calls=0 tl.ctx=0x7f417005cd70 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    117 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x800 umask=0x0 prmsk=0x800 pwmsk=0x0 owner=0x7f41540bf020 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=34418 rport=443 sv=api-splunk-tst-wus2-ahcrs01a/api-splunk-tst-wus2-ahcrs01a0 mux=PASS ctx=0x7f415406d6f0 xprt=SSL xprt_ctx=0x7f41540578b0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4154061428(ev=1 tl=0x7f415405c510 tl.calls=0 tl.ctx=0x7f4154061410 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    119 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x10 umask=0x0 prmsk=0x10 pwmsk=0x0 owner=0x7f414c0565d0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=57778 rport=443 sv=ui.control-tower.global.amadeus.net/ui.control-tower.global.amadeus.net0 mux=PASS ctx=0x7f414c086b60 xprt=SSL xprt_ctx=0x7f414c1125a0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f414c05e0e8(ev=1 tl=0x7f414c113090 tl.calls=0 tl.ctx=0x7f414c05e0d0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    122 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x200 umask=0x0 prmsk=0x200 pwmsk=0x200 owner=0x7f416018d570 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39618 sv=portal-fra/portal-fra0 mux=PASS ctx=0x7f416007a510 xprt=SSL xprt_ctx=0x7f416004ddf0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f416018d3c8(ev=1 tl=0x7f4160059540 tl.calls=0 tl.ctx=0x7f416018d3b0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    137 : st=0x010421(cL heOpi W:sRa R:srA) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x0 owner=0x7f4158109b80 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18800300 cerr=0 fam=ipv4 lport=38572 rport=443 sv=payproxy-dev/payproxy-dev1 mux=PASS ctx=0x7f415805c9c0 xprt=SSL xprt_ctx=0x7f415810de60 xctx.st=0 .err=0 .xprt=RAW .wait.ev=1 .subs=0x7f4158108808(ev=1 tl=0x7f4158065e80 tl.calls=0 tl.ctx=0x7f41581087f0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    141 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x1 owner=0x55d049d7fdd0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=50686 sv=ccb-amer/ccb-amer0 mux=PASS ctx=0x55d049bd3ea0 xprt=SSL xprt_ctx=0x55d049c3edf0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x55d049c5f688(ev=1 tl=0x55d049d84710 tl.calls=0 tl.ctx=0x55d049c5f670 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    157 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x2000 owner=0x7f41640da100 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39636 sv=alf-fra/alf-fra0 mux=PASS ctx=0x7f416405af40 xprt=SSL xprt_ctx=0x7f4164090150 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41640b2328(ev=1 tl=0x7f41640e8b00 tl.calls=0 tl.ctx=0x7f41640b2310 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    174 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x400 umask=0x0 prmsk=0x400 pwmsk=0x400 owner=0x7f4158104bf0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=35820 sv=uatcem.cb/uatcem.cb0 mux=PASS ctx=0x7f41581ac220 xprt=SSL xprt_ctx=0x7f415806a5a0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415806cd28(ev=1 tl=0x7f41581ac9d0 tl.calls=0 tl.ctx=0x7f415806cd10 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    179 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x10 umask=0x0 prmsk=0x10 pwmsk=0x10 owner=0x7f414c058230 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=60588 sv=grafana.ngi-prd-euw3b-c-39.ngi-box-prd.gce/grafana.ngi-prd-euw3b-c-39.ngi-box-prd.gce0 mux=PASS ctx=0x7f414c052720 xprt=SSL xprt_ctx=0x7f414c0a0d90 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f414c05b188(ev=1 tl=0x7f414c101680 tl.calls=0 tl.ctx=0x7f414c05b170 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    184 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x40 umask=0x0 prmsk=0x40 pwmsk=0x40 owner=0x7f41500a58d0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39614 sv=uniping-config-fra/uniping-config-fra0 mux=PASS ctx=0x7f41501868d0 xprt=SSL xprt_ctx=0x7f4150072450 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415005cb88(ev=1 tl=0x7f41500f4a60 tl.calls=0 tl.ctx=0x7f415005cb70 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    196 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x200 umask=0x0 prmsk=0x200 pwmsk=0x200 owner=0x7f4160085680 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=42976 sv=ad-reef-di-rect-prd.muc6/ad-reef-di-rect-prd.muc60 mux=PASS ctx=0x7f41601d61a0 xprt=SSL xprt_ctx=0x7f41600773e0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41601bf208(ev=1 tl=0x7f4160071c90 tl.calls=0 tl.ctx=0x7f41601bf1f0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    212 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1 umask=0x0 prmsk=0x1 pwmsk=0x1 owner=0x55d049c3e0a0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39620 sv=workbench-fra3/workbench-fra30 mux=PASS ctx=0x55d049b7c8a0 xprt=SSL xprt_ctx=0x55d049bcd180 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x55d049d020d8(ev=1 tl=0x55d049c2bb40 tl.calls=0 tl.ctx=0x55d049d020c0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    214 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x4 owner=0x7f41700c88a0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39638 sv=fra3-api-sre-self-services/fra3-api-sre-self-services0 mux=PASS ctx=0x7f417017bb70 xprt=SSL xprt_ctx=0x7f4170059e00 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f417004dea8(ev=1 tl=0x7f417007b2b0 tl.calls=0 tl.ctx=0x7f417004de90 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    230 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x1000 umask=0x0 prmsk=0x1000 pwmsk=0x1000 owner=0x7f415c076470 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=39642 sv=topodb-fra/topodb-fra0 mux=PASS ctx=0x7f415c063530 xprt=SSL xprt_ctx=0x7f415c065810 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f415c05f7f8(ev=1 tl=0x7f415c0503a0 tl.calls=0 tl.ctx=0x7f415c05f7e0 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    237 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x8 umask=0x0 prmsk=0x8 pwmsk=0x8 owner=0x7f41680f6e80 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=47668 sv=oncalldb-fra/oncalldb-fra0 mux=PASS ctx=0x7f416804de70 xprt=RAW
    241 : st=0x010011(cL heopi W:srA R:srA) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x4 owner=0x7f41700ca010 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x10400300 cerr=0 fam=ipv4 lport=48704 sv=lbe-sip-tst-admin2/lbe-sip-tst-admin20 mux=PASS ctx=0x7f417007a6e0 xprt=RAW
    262 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x2000 umask=0x0 prmsk=0x2000 pwmsk=0x2000 owner=0x7f4164084ba0 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=57438 sv=boxapi.ngi-prd-euw3b-c-39.ngi-box-prd.gce/boxapi.ngi-prd-euw3b-c-39.ngi-box-prd.gce0 mux=PASS ctx=0x7f4164057a50 xprt=SSL xprt_ctx=0x7f416405fca0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f416404fe38(ev=1 tl=0x7f41640fd3f0 tl.calls=0 tl.ctx=0x7f416404fe20 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0
    284 : st=0x010010(cL heopi W:srA R:sra) ref=0 gid=1 tmask=0x4 umask=0x0 prmsk=0x4 pwmsk=0x4 owner=0x7f41540ffc00 iocb=0x55d045affab0(sock_conn_iocb) back=1 cflg=0x18c00300 cerr=0 fam=ipv4 lport=48790 sv=yopass.forge.amadeus.net/yopass.forge.amadeus.net0 mux=PASS ctx=0x7f4170198cf0 xprt=SSL xprt_ctx=0x7f41700623c0 xctx.st=0 .err=0 .xprt=RAW .wait.ev=2 .subs=0x7f41700f0e38(ev=1 tl=0x7f417007a160 tl.calls=0 tl.ctx=0x7f41700f0e20 tl.fct=srv_chk_io_cb) .sent_early=0 .early_in=0

show profiling tasks
Tasks activity:
  function                      calls   cpu_tot   cpu_avg   lat_tot   lat_avg
  ssl_sock_io_cb                32414   8.028s    247.7us   590.2ms   18.21us <- sock_conn_iocb@src/sock.c:875 tasklet_wakeup
  process_chk                   30032   2.497s    83.16us   432.4ms   14.40us <- wake_expired_tasks@src/task.c:350 task_wakeup
  process_chk                   27795   1.783s    64.15us   13.53ms   486.0ns <- wake_srv_chk@src/check.c:1087 task_wakeup
  ssl_sock_io_cb                25477   2.687s    105.5us   2.389s    93.77us <- ssl_sock_start@src/ssl_sock.c:5606 tasklet_wakeup
  srv_chk_io_cb                 24653   24.66ms   1.000us   954.7ms   38.72us <- ssl_sock_io_cb@src/ssl_sock.c:6272 tasklet_wakeup
  ssl_sock_io_cb                24384   1.001s    41.07us   393.2ms   16.12us <- sock_conn_iocb@src/sock.c:854 tasklet_wakeup
  process_chk                    1429   117.0ms   81.84us   64.84ms   45.38us <- run_tasks_from_lists@src/task.c:658 task_drop_running
  process_chk                     540   390.1us   722.0ns   6.239ms   11.55us <- wake_expired_tasks@src/task.c:435 task_drop_running
  process_chk                     451   9.769ms   21.66us   472.4us   1.047us <- wake_srv_chk@src/check.c:1080 task_wakeup
  process_stream                    4   348.6us   87.15us   2.440us   610.0ns <- sc_notify@src/stconn.c:1141 task_wakeup
  h1_io_cb                          4   101.1us   25.28us   19.26us   4.814us <- ssl_sock_io_cb@src/ssl_sock.c:6272 tasklet_wakeup
  accept_queue_process              2   30.27us   15.14us   83.94us   41.97us <- listener_accept@src/listener.c:1469 tasklet_wakeup
  process_stream                    2   269.3us   134.7us   70.22us   35.11us <- stream_new@src/stream.c:578 task_wakeup
  task_run_applet                   1   499.0ns   499.0ns   4.480us   4.480us <- sc_app_shut_applet@src/stconn.c:911 appctx_wakeup
  sc_conn_io_cb                     1   9.175us   9.175us   692.0ns   692.0ns <- h1_wake_stream_for_recv@src/mux_h1.c:2578 tasklet_wakeup
  h1_timeout_task                   1   1.013us   1.013us   30.68us   30.68us <- h1_release@src/mux_h1.c:1045 task_wakeup
  ssl_sock_io_cb                    1   300.0ns   300.0ns   2.075us   2.075us <- conn_subscribe@src/connection.c:736 tasklet_wakeup
  sc_conn_io_cb                     1   13.64us   13.64us   680.0ns   680.0ns <- sock_conn_iocb@src/sock.c:875 tasklet_wakeup
  sc_conn_io_cb                     1   37.57us   37.57us   23.12us   23.12us <- h1_wake_stream_for_send@src/mux_h1.c:2588 tasklet_wakeup
  sc_conn_io_cb                     1   1.431us   1.431us   5.635us   5.635us <- sc_app_chk_rcv_conn@src/stconn.c:780 tasklet_wakeup
  task_run_applet                   1   35.09us   35.09us   5.006us   5.006us <- sc_applet_create@src/stconn.c:502 appctx_wakeup

perf top
   6.16%  [kernel]             [k] sk_run_filter
   4.34%  perf                 [.] __symbols__insert
   4.18%  [kernel]             [k] _raw_spin_unlock_irqrestore
   2.96%  [kernel]             [k] __do_softirq
   2.04%  perf                 [.] rb_next
   1.93%  [kernel]             [k] vmxnet3_xmit_frame
   1.66%  [kernel]             [k] system_call_after_swapgs
   1.43%  [kernel]             [k] finish_task_switch
   1.37%  [kernel]             [k] format_decode
   1.30%  [kernel]             [k] run_timer_softirq
   1.16%  [kernel]             [k] number.isra.2
   1.11%  [kernel]             [k] vsnprintf
   0.84%  [kernel]             [k] ipt_do_table
   0.72%  [kernel]             [k] arch_cpu_idle
   0.68%  [kernel]             [k] rcu_process_callbacks
   0.67%  [kernel]             [k] __do_page_fault
   0.66%  [kernel]             [k] kallsyms_expand_symbol.constprop.1
   0.62%  [kernel]             [k] strnlen
   0.62%  libc-2.17.so         [.] _int_malloc
   0.61%  [kernel]             [k] string.isra.7
   0.54%  [kernel]             [k] __memcpy
   0.54%  [kernel]             [k] module_get_kallsym
   0.50%  [kernel]             [k] vmxnet3_poll_rx_only
   0.48%  libc-2.17.so         [.] _int_free
   0.45%  [kernel]             [k] _raw_qspin_lock
   0.44%  libpthread-2.31.so   [.] 0x0000000000012314
   0.44%  [kernel]             [k] clear_page
   0.43%  [kernel]             [k] nf_iterate
   0.40%  perf                 [.] rb_insert_color
   0.39%  [vdso]               [.] __vdso_clock_gettime
   0.39%  [kernel]             [k] nf_ct_delete
   0.38%  libc-2.17.so         [.] __strlen_sse2_pminub
   0.37%  [kernel]             [k] rcu_idle_exit

wtarreau commented 10 months ago

Well, looking at the stats indicates that haproxy is doing almost nothing compared to the system. Actually the non-kernel stuff (libc, libpthread) is called by openssl (_int_malloc, _int_free and __strlen* usually come from it). Your "show fd" shows that among 50 backend connections, 48 are dedicated to health checks.

The "show activity" indicates a smooth distribution across all your threads so there's nothing really odd there. "show info" shows the process is overall mostly idle (94%), so what I'm suspecting here is that the vast majority of the load solely comes from SSL computations related to health checks. Do you know if your servers employ heavy keys (e.g. RSA4096) or anything special maybe ?

You may try to check them less often by setting the "inter" parameter on your "default-server" line. E.g. "inter 5000" to perform a check every 5 seconds instead of every 2 seconds. I suspect you'll see the load divided by 2.5.

Another point, do you observe the load climb at reload and progressively go down ? I'm asking because the more threads you have, the more handshakes will be needed until all threads have a valid SSL session. In 2.9 we've permitted to share the SSL sessions between threads so that any single session is sufficient for all threads. It speeds up the convergence and lowers the overall CPU usage during the first minutes. If the load you're seeing is a problem for you (even though it's 6-10% total), you might be interested in this. However you'd need to use the latest snapshot for this.

lukastribus commented 10 months ago

There are about 2000 backends, so if health checking is on, that would be quite some traffic indeed.

Do you have more than one server per backend? There is really not much point in health checking if there is only a single server per backend.

mvgadagi commented 10 months ago

@wtarreau

I have checked and the majority of the backend they are using SHA-384 With RSA Encryption, SHA-256 With RSA Encryption
I have just tested the configuration with default-server init-addr last,libc,none inter 5000 but still I see that haproxy in htop shows more than 100%
Actually, the load seems to always be in the range of 110 % to 140% it keeps varying in the range, irrespective of the start and running of the haproxy service

I don't have a problem in the load more worry was there was some additional delay (~700ms - ~900ms )caused while servicing the traffic via proxy, So I thought since the proxy in the process is consuming more than 100% CPU, it could be bit overloaded and hence there was some delay.

Is there anyway to identify that haproxy is waiting for something before processing the request, I have looked the queue time in stats it was 0ms.

mvgadagi commented 10 months ago

@lukastribus Unfortunately, we have some 5% backend in which multiple servers are targeted. In most of the cases, it's a single server.

mvgadagi commented 10 months ago

@wtarreau , sincere apology. I was targeting the wrong config. After setting default-server init-addr last,libc,none inter 5000 the htop usages show just 49% now.

wtarreau commented 10 months ago

OK so it's clearly SSL health checks to all these servers that's responsible for the CPU usage. As Lukas said, if they're a concern, maybe some can be disabled (or just slowed down).

However, based on your "show activity" output, we're seeing in avg_loop_us that the average loop time is 135 microseconds, or 6000 times lower than the 900ms delay you're seeing, and the highest reported latency in "show task" is even smaller at 93 microseconds, so I'm afraid the delays come from something else.

It could possibly come from Lua processing if you're requesting external information for each request (though 900ms is a bit extreme).

mvgadagi commented 10 months ago

Regarding the Lua processing, after working with @lukastribus , I have removed all the forbidden stuff from Lua as mentioned here https://github.com/haproxy/haproxy/blob/master/doc/lua.txt#L161 So now I am a bit confident that it's something other than proxy.

Thanks, @wtarreau @lukastribus for your help and for making it very clear.

git001 commented 10 months ago

@lukastribus Unfortunately, we have some 5% backend in which multiple servers are targeted. In most of the cases, it's a single server.

If the backend are on the same server could track help to reduce the health checks.

With track can you define that server health are checked from one health check and the other server relay on that.

Here a snippet from my setup which reduced the check traffic.

backend my_backend
...
server live_fail server_01:443 ... track be_live_failover/live_fail

backend be_live_failover
  log global

  option httpchk
  http-check connect ssl sni {{ hosts["live_failover"] }}
  http-check send meth GET uri {{ http_checks["live_http"]}} ver HTTP/1.1 hdr host {{ hosts["live_failover"]}}

  http-request set-header Host {{ hosts["live_failover"] }}

  server live_fail server_01:443 resolvers mydns ssl check check-ssl check-sni {{ hosts["live_failover"] }} sni str({{ hosts["live_failover"] }}) ca-file /etc/haproxy/letsencryptauthorityx3.pem

haproxy / haproxy