syslog-ng-ctl reload is taking more time

[deepasoni27]

syslog-ng

Version of syslog-ng

3.26.1

Platform

ubuntu 4.19.94-rt39

Debug bundle

Issue

syslog-ng strace :45:01.551907 fcntl64(42, F_SETFD, FD_CLOEXEC) = 0 <0.000041> 04:45:01.552065 setsockopt(42, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0 <0.000044> 04:45:01.552291 fcntl64(42, F_GETFD) = 0x1 (flags FD_CLOEXEC) <0.000043> 04:45:01.552444 fcntl64(42, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) <0.000039> 04:45:01.552579 setsockopt(42, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 <0.000042> 04:45:01.552735 accept(20, 0xbefc45a8, [1024]) = -1 EAGAIN (Resource temporarily unavailable) <0.000062> 04:45:01.552915 epoll_ctl(5, EPOLL_CTL_ADD, 42, {EPOLLIN, {u32=761756, u64=1546188988316}}) = 0 <0.000046> 04:45:01.553074 epoll_wait(5, [{EPOLLIN|EPOLLHUP, {u32=761756, u64=1546188988316}}], 22, -1) = 1 <0.000042> 04:45:01.553230 epoll_ctl(5, EPOLL_CTL_DEL, 42, 0xbefc4a18) = 0 <0.000043> 04:45:01.553370 epoll_wait(5, [{EPOLLIN, {u32=167048, u64=13140747368926448776}}], 22, -1) = 1 <51.444811> 04:45:52.998369 epoll_ctl(5, EPOLL_CTL_DEL, 9, 0xbefc4940) = 0 <0.000047> 04:45:52.998543 close(9) = 0 <0.000048> 04:45:52.998734 epoll_ctl(5, EPOLL_CTL_DEL, 39, 0xbefc4920) = 0 <0.000045> 04:45:52.998886 epoll_ctl(5, EPOLL_CTL_DEL, 37, 0xbefc4920) = 0 <0.000044>

$ gdb syslog-ng run -->

Steps to reproduce

make all entries in /etc/resolv.conf unreachable and to syslog-ng.conf add http based destination

Configuration

http(url() method("POST") tls( ca-file() cert-file() key-file(>FILE_PATH>) peer-verify(yes) ) disk-buffer( mem-buf-size(50000) disk-buf-size(20000000) reliable(yes) dir("/extra/logs/syslog-disk-buf") ) batch-lines(5) batch-bytes(5000) batch-timeout(10000) persist-name("syslog-http-destination") headers("Content-Type: application/x-ndjson") body-suffix("\n") body('{"index":{}} $(format-json --pair DEVICEID=S147T1211700151 --pair SYSLOG-MSG=$(template descpri))')

When every we reload config using "syslog-ng-ctl reload" it is taking random time best 2-3sec to worst more than 120sec .At server strace we saw it is stuck at epoll_wait.

Please confirm if this is known issue or fixed on later, or some configuration is required to handle http based destination when DNS is unreachable.

Thanks a lot!

Input and output logs (if possible)

[deepasoni27]

I am seeing sometimes reloading taking more time , say 30sec when DNS is reachable also.

[deepasoni27]

Please check if any help on this ASAP.

Thanks a lot!

[Kokan]

Does the shutdown is also slow ? (syslog-ng-ctl stop) In that case I would suspect maybe it is related to your http() destination. By default the http() destination blocks (reload/shutdown) till it can perform the operation. (There is a timeout() option to control this.) If that does not help, if possible please provide a backtrace when it hangs for a long time and logs from syslog-ng preferrebly with debug and trace on (-dvt).

[deepasoni27]

Thanks for quick update. Yes it is only happening with http destination. You are correct shutdown also taking more time. Do I need to add timeout under http destination?

[Kokan]

It really depends on you. That parameter configures the http destination as such that if the sending of logs took more than timeout() time, it aborts that operation. Hence quicker reload on the cost of aborted log transfer. On the other hand a http operation may block for a long long time.

[deepasoni27]

Even with bad URL where logs are buffered and not sent. I am see delay upto 120seconds and more

I will share backtrace as well and debug logs.Thanks

[deepasoni27]

I tried with shutting down all syslog clients as well. So nothing was logged still reload was getting stuck mostly when none of DNS are reachable. Normally DNS lookup should take 5 seconds.