search

Teradata / PyTd

A Python Module to make it easy to script powerful interactions with Teradata Database in a DevOps friendly way.
MIT License
108 stars 43 forks source link

malloc(): memory corruption (fast) with TTU 15.10 but not with 14.10 and 16.00 #81

Open ericsun2 opened 6 years ago

ericsun2 commented 6 years ago

While testing the udaExec.connect(method='odbc', system='dwhdev', username='ETL_TEST', password='password') we encounter the following malloc error.

When we switch to TTU 14.10 and 16.00 on different hosts to isolate the issue, we realized that the issue only happened to TTU 15.10 

2017-10-09 23:19:10,552 - teradata.udaexec - INFO - Execution Details:
/********************************************************************************
 * Application Name: QuickTest
 *          Version: 0.1
 *       Run Number: 20171009231910-3
 *             Host: etl01.test.foo.com
 *         Platform: Linux-2.6.32-431.17.1.el6.x86_64-x86_64-with-redhat-6.5-Santiago
 *          OS User: data_test
 *   Python Version: 3.6.1
 *  Python Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-11)
 *     Python Build: ('default', 'Apr 19 2017 21:58:30')
 *  UdaExec Version: 15.10.0.21
 *     Program Name: td_test.py
 *      Working Dir: /var/tmp/test
 *          Log Dir: /var/tmp/test/logs
 *         Log File: /var/tmp/test/logs/QuickTest.20171009231910-3.log
 *     Config Files: ['/etc/udaexec.ini: Not Found', '/export/home/data_test/udaexec.ini: Not Found', '/var/tmp/test/udaexec.ini: Not Found']
 *      Query Bands: ApplicationName=QuickTest;Version=0.1;JobID=20171009231910-3;ClientUser=data_test;Production=False;udaAppLogFile=/var/tmp/test/logs/QuickTest.20171009231910-3.log;UtilityName=PyTd;UtilityVersion=15.10.0.21
********************************************************************************/
2017-10-09 23:19:10,556 - teradata.udaexec - INFO - Creating connection: {'method': 'odbc', 'system': 'dwhdev', 'username': 'ETL_TEST', 'password': 'XXXXXX'}
2017-10-09 23:19:10,556 - teradata.tdodbc - INFO - Loading ODBC Library: /opt/teradata/client/ODBC_64/lib/libodbc.so
2017-10-09 23:19:10,828 - teradata.tdodbc - INFO - Available drivers: Teradata, 
*** glibc detected *** /export/apps/python/3.6/bin/python3: malloc(): memory corruption (fast): 0x000000000191a7a0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ead876166]
/lib64/libc.so.6[0x3ead879f2f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x3ead87a9a1]
/lib64/libc.so.6(catopen+0x3c)[0x3ead83111c]
/opt/teradata/client/ODBC_64/lib/tdata.so(OS_Catopen+0xc9)[0x7f23713db4f9]
/opt/teradata/client/ODBC_64/lib/tdata.so(_Z8AllocEnvPPv+0x124)[0x7f23712f8284]
/opt/teradata/client/ODBC_64/lib/tdata.so(SQLAllocHandle+0x222)[0x7f23712fa1c2]
/opt/teradata/client/ODBC_64/lib/libodbc.so(+0xb7c6b)[0x7f2372cbec6b]
/opt/teradata/client/ODBC_64/lib/libodbc.so(+0xb78f1)[0x7f2372cbe8f1]
/opt/teradata/client/ODBC_64/lib/libodbc.so(+0xab9f5)[0x7f2372cb29f5]
/opt/teradata/client/ODBC_64/lib/libodbc.so(+0xa4e73)[0x7f2372cabe73]
/opt/teradata/client/ODBC_64/lib/libodbc.so(SQLDriverConnectW+0x149)[0x7f2372c95163]
/usr/lib64/libffi.so.5(ffi_call_unix64+0x4c)[0x7f2375138dac]
/usr/lib64/libffi.so.5(ffi_call+0x214)[0x7f2375138b34]
/export/apps/python/3.6/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x3dc)[0x7f237534bdec]
/export/apps/python/3.6/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9353)[0x7f2375344353]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x9e)[0x7f237c649d4e]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0x14855b)[0x7f237c72655b]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x26fd)[0x7f237c7293fd]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0x146d00)[0x7f237c724d00]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c)[0x7f237c724f9c]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166)[0x7f237c649e16]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc)[0x7f237c64a06c]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyObject_Call+0x56)[0x7f237c64a106]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0xe44c3)[0x7f237c6c24c3]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0xd9fdc)[0x7f237c6b7fdc]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyObject_Call+0x56)[0x7f237c64a106]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x37d6)[0x7f237c72a4d6]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0x146d00)[0x7f237c724d00]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0x14872a)[0x7f237c72672a]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1e7d)[0x7f237c728b7d]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(+0x146d00)[0x7f237c724d00]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63)[0x7f237c725183]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyEval_EvalCode+0x3b)[0x7f237c7251cb]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyRun_FileExFlags+0x130)[0x7f237c757840]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(PyRun_SimpleFileExFlags+0x173)[0x7f237c758e03]
/export/apps/python/3.6.1/lib/libpython3.6m.so.1.0(Py_Main+0xe21)[0x7f237c774231]
/export/apps/python/3.6/bin/python3(main+0x16d)[0x400b7d]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3ead81ed1d]
/export/apps/python/3.6/bin/python3[0x400949]
======= Memory map: ========
00400000-00401000 r-xp 00000000 09:03 35800514                           /export/apps/python/3.6.1/bin/python3.6
00601000-00602000 rw-p 00001000 09:03 35800514                           /export/apps/python/3.6.1/bin/python3.6
016c9000-0193f000 rw-p 00000000 00:00 0                                  [heap]
3192e00000-3192e15000 r-xp 00000000 09:03 36180865                       /export/apps/data_svc/lib64/libz.so.1.2.3
...

Don't see any obvious issue with TTU installation.

$ ldd /opt/teradata/client/ODBC_64/lib/tdata.so
    linux-vdso.so.1 =>  (0x00007fffe1fff000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f396565f000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3965449000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f396522b000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3965027000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f3964e1f000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f3964c05000)
    libodbcinst.so => /usr/lib64/libodbcinst.so (0x00007f39649d6000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f3964752000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f39643bd000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003372200000)
    libddicu27.so => /usr/lib64/libddicu27.so (0x00007f39634c6000)

$ ls -l /usr/lib64/libodbc.so /usr/lib64/libodbcinst.so
lrwxrwxrwx 1 root root 47 Nov  2  2015 /usr/lib64/libodbcinst.so -> /opt/teradata/client/15.10/lib64/libodbcinst.so
lrwxrwxrwx 1 root root 43 Nov  2  2015 /usr/lib64/libodbc.so -> /opt/teradata/client/15.10/lib64/libodbc.so

Please advise how to resolve this malloc problem.

escheie commented 6 years ago

I found a matching ODBC bug for this issue (ODBC-17044). The bug is fixed in ODBC driver 15.10.01.004. What version are you using?

ericsun2 commented 6 years ago 
$ grep -a 15.10 tdata.so
15.10.00.00     Wrap Count = %d
CNFP_VERSION=15.10.00.00

Let me try to get the latest http://downloads.teradata.com/download/license?destination=download/files/7455/200430/0/tdodbc1510__linux_indep.15.10.01.06-1.tar.gz and give it a shot.

escheie commented 6 years ago

@ericsun2 Is this issue resolved for you?