omiagent keeps crashing

[johanburati]

omiagent keeps crashing and filling up the file system with core files.

OMI version

/opt/omi/bin/omiagent: OMI-1.6.2-0 - Wed May 29 21:21:54 PDT 2019

Packages version

auoms-2.0.0-13.x86_64
dsc-1.1.1-294.x86_64
omi-1.6.2-0.x86_64
omsagent-1.11.0-7.x86_64
omsconfig-1.1.1-926.x86_64
scx-1.6.3-659.x86_64

Core file

[New LWP 7765]
[New LWP 4710]
[New LWP 4784]
[New LWP 4786]
[New LWP 4717]
[New LWP 4785]
[New LWP 4673]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/omi/bin/omiagent 9 10 --destdir / --providerdir /opt/omi/lib --loglevel WA'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f9dc7112e00 in ?? ()
Missing separate debuginfos, use: debuginfo-install omi-1.6.2-0.x86_64
#0  0x00007f9dc7112e00 in ?? ()
#1  0x00007f9dc7156f47 in ?? ()
#2  0x00007f9dc7156b60 in ?? ()
#3  0xffffffff00000073 in ?? ()
#4  0x0000000000000000 in ?? ()
From                To                  Syms Read   Shared Object Library
0x00007f9dd29cb8b0  0x00007f9dd29d6d51  Yes (*)     /lib64/libpthread.so.0
0x00007f9dd27c2d90  0x00007f9dd27c388e  Yes (*)     /lib64/libdl.so.2
0x00007f9dd25b5670  0x00007f9dd25bcd0a  Yes (*)     /lib64/libpam.so.0
0x00007f9dd235bb00  0x00007f9dd23975ad  Yes (*)     /opt/omi/lib/libssl.so.1.0.0
0x00007f9dd1f4c000  0x00007f9dd20845c7  Yes (*)     /opt/omi/lib/libcrypto.so.1.0.0
0x00007f9dd1b318d0  0x00007f9dd1c80baf  Yes (*)     /lib64/libc.so.6
0x00007f9dd2be2ad0  0x00007f9dd2bfdf80  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007f9dd18ec240  0x00007f9dd18f27f2  Yes (*)     /lib64/libaudit.so.1
0x00007f9dd16a8060  0x00007f9dd16da50a  Yes (*)     /lib64/libgssapi_krb5.so.2
0x00007f9dd13d8b00  0x00007f9dd14409ba  Yes (*)     /lib64/libkrb5.so.3
0x00007f9dd11b0500  0x00007f9dd11b10d3  Yes (*)     /lib64/libcom_err.so.2
0x00007f9dd0f80890  0x00007f9dd0f9eb7f  Yes (*)     /lib64/libk5crypto.so.3
0x00007f9dd0d68110  0x00007f9dd0d74698  Yes (*)     /lib64/libz.so.1
0x00007f9dd0b61580  0x00007f9dd0b633bc  Yes (*)     /lib64/libcap-ng.so.0
0x00007f9dd0953aa0  0x00007f9dd095b21b  Yes (*)     /lib64/libkrb5support.so.0
0x00007f9dd074d5b0  0x00007f9dd074e1cc  Yes (*)     /lib64/libkeyutils.so.1
0x00007f9dd05368f0  0x00007f9dd0544e52  Yes (*)     /lib64/libresolv.so.2
0x00007f9dd03129d0  0x00007f9dd03287d6  Yes (*)     /lib64/libselinux.so.1
0x00007f9dd00ab5f0  0x00007f9dd00f15b0  Yes (*)     /lib64/libpcre.so.1
0x00007f9dcfc64c00  0x00007f9dcfdc2898  Yes (*)     /opt/omi/lib/libSCXCoreProviderModule.so
0x00007f9dcf900810  0x00007f9dcf91b198  Yes (*)     /opt/omi/lib/libmicxx.so
0x00007f9dcf6c4db0  0x00007f9dcf6c9c6c  Yes (*)     /lib64/libcrypt.so.1
0x00007f9dcf4be1e0  0x00007f9dcf4c116c  Yes (*)     /lib64/librt.so.1
0x00007f9dcf210220  0x00007f9dcf2772da  Yes (*)     /lib64/libstdc++.so.6
0x00007f9dceeb8310  0x00007f9dcef232d6  Yes (*)     /lib64/libm.so.6
0x00007f9dcec9fa90  0x00007f9dcecaf245  Yes (*)     /lib64/libgcc_s.so.1
0x00007f9dcea9ab40  0x00007f9dcea9b1bb  Yes (*)     /lib64/libfreebl3.so
0x00007f9dc835f130  0x00007f9dc8366481  Yes (*)     /lib64/libnss_files.so.2
0x00007f9dc8157020  0x00007f9dc815a87f  Yes (*)     /lib64/libnss_dns.so.2
0x00007f9dc7f42dc0  0x00007f9dc7f4b21a  Yes (*)     /lib64/libnss_myhostname.so.2
0x00007f9dc7d3d620  0x00007f9dc7d3ee56  Yes (*)     /lib64/libcap.so.2
0x00007f9dc7af6c80  0x00007f9dc7b2cca3  Yes (*)     /lib64/libdw.so.1
0x00007f9dc78e93d0  0x00007f9dc78eb40a  Yes (*)     /lib64/libattr.so.1
0x00007f9dc76d2f90  0x00007f9dc76e23d3  Yes (*)     /lib64/libelf.so.1
0x00007f9dc74ad080  0x00007f9dc74c3860  Yes (*)     /lib64/liblzma.so.5
0x00007f9dc729b760  0x00007f9dc72a75f0  Yes (*)     /lib64/libbz2.so.1
0x00007f9dc65102e0  0x00007f9dc653f2ef  Yes (*)     /lib64/libssl3.so
0x00007f9dc62e62c0  0x00007f9dc62fad97  Yes (*)     /lib64/libsmime3.so
0x00007f9dc5fc9df0  0x00007f9dc609efd4  Yes (*)     /lib64/libnss3.so
0x00007f9dc5d8c990  0x00007f9dc5d9c698  Yes (*)     /lib64/libnssutil3.so
0x00007f9dc5b7cec0  0x00007f9dc5b7dc28  Yes (*)     /lib64/libplds4.so
0x00007f9dc59784b0  0x00007f9dc5979b16  Yes (*)     /lib64/libplc4.so
0x00007f9dc5745c30  0x00007f9dc5765eaf  Yes (*)     /lib64/libnspr4.so
0x00007f9da75c5280  0x00007f9da75eefac  Yes (*)     /lib64/libsoftokn3.so
0x00007f9da7316a00  0x00007f9da739e860  Yes (*)     /lib64/libsqlite3.so.0
0x00007f9da7086600  0x00007f9da70e4a6d  Yes (*)     /lib64/libfreeblpriv3.so
0x00007f9da6e80c70  0x00007f9da6e8136b  Yes (*)     /lib64/libnsssysinit.so
0x00007f9da6c587c0  0x00007f9da6c72bb9  Yes (*)     /lib64/libnsspem.so
(*): Shared library is missing debugging information.

[johanburati]

I could trace the issue to ConsistencyInvoker, it is called every 15 min from a cron job and each time it runs, omiagent crash with a segfault.

I have opened an issue in the DSC repo as well (https://github.com/microsoft/PowerShell-DSC-for-Linux/issues/601)

[JumpingYang001]

@johanburati could you file an ICM and attach the core file in ICM and assign it to omi team? BTW, what is your Operating System version? It is not dsc issue, dsc don't use omi after 1.1.1 versoin, so you can close dsc issue.

[johanburati]

@JumpingYang001, I have already created a ICM on 2019-07-23 with a link to the core files, it was assigned to the Omi team and you already commented on it, but there is no update since.

I would appreciate if someone could follow up on it, let me know if you need me to send you the number.

[JumpingYang001]

@johanburati yes, I need the ICM number, thanks.

[johanburati]

@JumpingYang001 135138389

[johanburati]

For reference it was a dsc issue, the DSC extension installs version 1.1.1-294 of the dsc package, this version cause omiagent to segfault, installing version 1.1.1-926 manually fix the issue.

I have submitted a fix for the extension https://github.com/Azure/azure-linux-extensions/pull/877

[JumpingYang001]

Since it is not omi issue, I close this issue, thanks.

[johanburati]

If you are having this issue check https://github.com/Azure/azure-linux-extensions/issues/875 for details and solution.