driving kopano-dopcker mad : P

[osnet]

hi @mlan

how much RAM did you assin to kopano / php ?

i get this with 10 Users EAS incl Publicfolders

app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Fatal error detected. Please report all following information.
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: kopano-dagent 10.0.6
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: OS: Ubuntu 18.04.5 LTS (Linux 4.18.0-240.1.1.el8_3.x86_64 x86_64)
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Thread name: kopano-dagent
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Peak RSS: 36440
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Pid 1441 caught SIGABRT (6), out of memory or unhandled exception, traceback:
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Backtrace:
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x4cb40) [0x7f3e46e5db40]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x320a6) [0x7f3e46e430a6]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x350cd) [0x7f3e46e460cd]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f3e4699d980]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f3e45a7bfb7]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f3e45a7d921]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f6. /lib/x86_64-linux-gnu/libc.so.6(+0x89967) [0x7f3e45ac6967]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f7. /lib/x86_64-linux-gnu/libc.so.6(+0x134b8f) [0x7f3e45b71b8f]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f8. /lib/x86_64-linux-gnu/libc.so.6(+0x134bb1) [0x7f3e45b71bb1]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f9. /lib/x86_64-linux-gnu/libc.so.6(+0x1328a0) [0x7f3e45b6f8a0]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f10. /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x105) [0x7f3e45b6f055]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f11. /lib/x86_64-linux-gnu/libc.so.6(__snprintf_chk+0x85) [0x7f3e45b6ef25]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f12. /usr/sbin/kopano-dagent(+0x139d4) [0x5592fee989d4]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f13. /usr/sbin/kopano-dagent(+0x14de9) [0x5592fee99de9]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f14. /usr/sbin/kopano-dagent(+0x173c1) [0x5592fee9c3c1]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f15. /usr/sbin/kopano-dagent(+0x19809) [0x5592fee9e809]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f16. /usr/sbin/kopano-dagent(+0x1bf56) [0x5592feea0f56]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f17. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC18unix_fork_functionEPFPvS0_ES0_iPi+0x6e) [0x7f3e46e5a80e]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f18. /usr/sbin/kopano-dagent(+0x1da00) [0x5592feea2a00]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f19. /usr/sbin/kopano-dagent(+0xc6a2) [0x5592fee916a2]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f20. /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f3e45a5ebf7]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: f21. /usr/sbin/kopano-dagent(+0xe51a) [0x5592fee9351a]
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Signal errno: Success, signal code: -6
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Sender pid: 1441, sender uid: 999, si_status: 0
app_1     | Jan 11 17:36:34 kopano-dagent[1441]: Signal value: 0, faulting address: 0x3e7000005a1
app_1     | *** buffer overflow detected ***: /usr/sbin/kopano-dagent terminated

[osnet]

i try to enlarge some thingys and provide feedback

[mlan]

Hi @osnet

One guess is that the plugin don't release all memory it uses, so that with each call more and more memory is claimed. I will see if i can run the code though a profiler.

[mlan]

I did some profiling and did not immediately see an issue. So, perhaps it is not related to the plugin after all? I note that if you have many simultaneously incoming emails the kopano-dagent spawn many processes to cope with the work load that might consume significant amounts of memory.

I used the demo within the repo to stress test the delivery components. From within the demo I run this:

for x in $(seq 99); do make mta-test_public >/dev/null; done

I did not see any issues during these stress tests. During the test, memory consumption increased hundreds of MB but came back down afterwards.

[osnet]

i recognized some interesting facts.

• since blocking failed attempts on sasl with fail2ban the dagent is more chilled
• setting virtual_transport = lmtp:inet:[111.222.333.444]:2003 instead of virtual_transport = lmtp:111.222.333.444:2003 things are more smooth in delivery
• some how the error occurs only if person X with out-of-office ( which does not work) gets an mail.
• postfix then is displaying status deferred ( lost conenction to 111.222.333.444 ....

i need more diggin into it

[osnet]

allright. this error fired up with enabled out of office . out of office message wont get sent original email wont be delivered to inbox

thats annoying somehow

[osnet]

the image 3 weeks ago was fine

[mlan]

Ok, so you are closing in on finding the cause of the issue. Do you think it is within the kopano-core components themselves or do you think it was something that was introduced in the docker image?

[osnet]

ya, there is a bugf or a faulty thin in the copano community core

do yourself a favor : )

all the stuff we are adding are working, but when u build a new image for dockerhub , spin it up, and test out of office.

if you get the reply, great, if you get

*** buffer overflow detected ***: /usr/sbin/kopano-dagent terminated
app_1     | Jan 14 14:01:42 kopano-dagent[339]: ----------------------------------------------------------------------
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Fatal error detected. Please report all following information.
app_1     | Jan 14 14:01:42 kopano-dagent[339]: kopano-dagent 10.0.6
app_1     | Jan 14 14:01:42 kopano-dagent[339]: OS: Ubuntu 18.04.5 LTS (Linux 3.10.0-1160.11.1.el7.x86_64 x86_64)
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Thread name: kopano-dagent
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Peak RSS: 15624
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Pid 339 caught SIGABRT (6), out of memory or unhandled exception, traceback:
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Backtrace:
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x4cb40) [0x7f5f7d5a0b40]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x320a6) [0x7f5f7d5860a6]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x350cd) [0x7f5f7d5890cd]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f5f7d0e0980]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f5f7c1befb7]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f5f7c1c0921]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f6. /lib/x86_64-linux-gnu/libc.so.6(+0x89967) [0x7f5f7c209967]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f7. /lib/x86_64-linux-gnu/libc.so.6(+0x134b8f) [0x7f5f7c2b4b8f]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f8. /lib/x86_64-linux-gnu/libc.so.6(+0x134bb1) [0x7f5f7c2b4bb1]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f9. /lib/x86_64-linux-gnu/libc.so.6(+0x1328a0) [0x7f5f7c2b28a0]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f10. /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x105) [0x7f5f7c2b2055]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f11. /lib/x86_64-linux-gnu/libc.so.6(__snprintf_chk+0x85) [0x7f5f7c2b1f25]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f12. /usr/sbin/kopano-dagent(+0x139d4) [0x55a81b01a9d4]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f13. /usr/sbin/kopano-dagent(+0x14de9) [0x55a81b01bde9]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f14. /usr/sbin/kopano-dagent(+0x173c1) [0x55a81b01e3c1]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f15. /usr/sbin/kopano-dagent(+0x19809) [0x55a81b020809]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f16. /usr/sbin/kopano-dagent(+0x1bf56) [0x55a81b022f56]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f17. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5f7d0d56db]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: f18. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f5f7c2a171f]
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Signal errno: Success, signal code: -6
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Sender pid: 339, sender uid: 999, si_status: 0
app_1     | Jan 14 14:01:42 kopano-dagent[339]: Signal value: 0, faulting address: 0x3e700000153
app_1     | Jan 14 14:01:43 kopano-dagent[405]: Starting kopano-dagent version 10.0.6 (pid 405 uid 0) (LMTP mode)
app_1     | Jan 14 14:01:43 kopano-dagent[405]: Starting kopano-dagent version 10.0.6 (pid 405 uid 999) (LMTP mode)

the responder thingy fucks up dagent : )

[mlan]

Yes, the OOF part of kopano-dagent from nightly builds is currently broken. Steps to reproduce the issue:

1. Activate OOF for a user kopano-oof -u demo -m 1 -t "Dunno when I return"
2. Send email to demo user.

[mlan]

@osnet

This issue has now been resolved by upstream. The latest docker image should be good now.