Closed alitheg closed 7 years ago
There was a packaging problem with 4.0.0 so I deleted it to reupload, but PyPI doesn't allow that. I majorly screwed up! v4.0.1 was supposed to be only minor safe issues. I'm looking at it now.
Are you selecting decimal/numeric values from the DB that are returned as decimal.Decimal objects? I fished a segfault for that on the fix-126 branch. If you are using decimal, can you test with that branch?
I plan on releasing it soon - today if possible, but was waiting for confirmation from the reporter.
On further review, there were no code changes between 4.0.0 and 4.0.1. I updated the docs and removed an unused function from setup.py.
Can you provide some more information about the segfault? Thanks.
v4.0.2 has been officially released. Can you test with it, please?
I had looked at the diffs between 4.0.0 and 4.0.1 and noticed there were no changes! I didn't understand how they could cause such different behaviour!
So the reads were working fine - I opened the page of my web app and all the information pulled through. But as soon as I tried to call the first endpoint that writes, the uWSGI process hosting my app segfaulted immediately. We're not using any decimal datatypes - it's a few ints and then a load of varchars.
I'm pushing a change this morning so I can test 4.0.2. It is working with 3.1.1, and the client is pushing me on a deadline, so I might have to leave it on 3.1.1 but I'll give it a quick test this morning.
So I updated to 4.0.2 and had the same issue - I'm wondering if I just got lucky with 4.0.0 for a while, and that perhaps I just have a problem with 4.x. The segfault is as below:
[pid: 8320|app: 0|req: 1/3] 10.58.1.60 () {48 vars in 1289 bytes} [Fri Jan 20 10:59:53 2017] GET /leaderboard/month => generated 1553 bytes in 224 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 0)
!!! uWSGI process 8318 got Segmentation Fault !!!
*** backtrace of 8318 ***
/var/investigate-api/.env/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46317e]
/var/investigate-api/.env/bin/uwsgi(uwsgi_segfault+0x21) [0x463511]
/usr/lib64/libc.so.6(+0x35250) [0x7f2237700250]
*** end of backtrace ***
DAMN ! worker 1 (pid: 8318) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 8385)
The first successful call is a read-only one, the call which then fails is one which writes.
I've reverted back to 3.1.1, as I can only reproduce this on the client's environment and we adopted pyodbc too late in the project to have time to debug it!
I fixed a double-free bug identified by the scan-build static analyzer that could have been at the root, but I doubt it. It only happens when you run out of memory trying to allocate Row. I did release v4.0.3 with this fix, so if you do every get a chance to try to reproduce this bug, please do so with 4.0.3.
Thanks.
@alitheg I know you reverted to an older version, but is there any chance you could test 4.0.3? If not I guess we can close this since it very well could have been the issue.
I should be able to use one of the dev environments to test it once the project is live - I'll take a look and see if I can reproduce again, I'll close if I can't or if I'm not able to - either way :).
And sorry for not noticing your 17-day old comment!
Do not close it. It is still happening. I've upgraded from 3.1.1 to 4.0.11 and had the same problem. I've also tried with 4.0.13 and same issue. Here's my report of the same problem
BTW, this fails on production only (I use Windows for dev and Linux on AWS for prod). If anyone guides me on how to provide debug information I definitely will as I can afford having the server down for a few minutes. @mkleehammer
We are seeing the same issue using pyodbc
with mod_wsgi
under Apache. Using pyodbc
< 4 still works. A colleague at work has the stack trace, but figured I'd mention we are seeing it too. I'll ping the colleague so they can provide more details. This has popped up when people install django-pyodbc-azure
because they install_requires
just provided a minimum pyodbc
version, so it grabs the latest / greatest version if requirements aren't pinned.
We're seeing this on 4.0.11 as well. Thanks, as always, for all of your fine efforts!
I'm not certain if the issue I'm seeing is this one, or if it is a separate issue. Using pyodbc 4.x via SQLAlchemy (I've tried pyodbc 4.0.1, 4.0.5, 4.0.11, and 4.0.14), executing a stored procedure (via session.execute()) results in my script crashing with one of:
This isn't a webapp, just a python script I'm running. The problem doesn't occur on every query, it seems to be anywhere from the 3rd to 6th or so (the script executes a stored procedure for each of many items). Executing the stored procedure manually doesn't show anything funny, and it's only returning about 11 rows.
I wouldn't have thought this was the same issue, except for the fact that downgrading pyodbc to 3.1.1 seems to completely resolve it. I've tried this on two different Ubuntu systems (16.10 running python 3.5 and 14.04 running python 3.4). I apologize if this isn't relevant to the issue, and it's very possible I'm overlooking a mistake on my part. I just thought I'd put this here on the off chance it's helpful.
Can anyone recommend the quickest way to reproduce this with the least amount of infrastructure. Also, has this been seen on macOS? If not, what abut Ubuntu 16?
The issue I am seeing is present on Ubuntu 16.10. Whether or not it is an example of this issue is something I'm still not completely certain of. I'll try to reproduce it under a simpler setup.
Well, I tried to reproduce my issue in a simple script using only pyodbc, and was not able to. I've come to the (tenuous) conclusion that I was doing something a little wonky in the SQLAlchemy layer (with engines and sessions). My experience may well have been unrelated to this particular issue.
After revamping my script to use connectionless execution in SQLAlchemy and pyodbc 4.0.14, I'm no longer having the problem. Sorry for any confusion my input caused, I had just wanted to share my results in case they provided any clues.
I'm trying to find a few hours to do a more solid repro (in the middle of a major product roll out at work), but here's what I've gotten so far:
On CentOS 6, with yum installed FreeTDS 0.91, we get a segfault with 4.0.12. On CentOS 7, with yum installed FreeTDS 0.95, we do not get a segfault with 4.0.12.
There are more variables in play that I have to investigate, but this one stuck out and I wasn't sure if anyone else had compared FreeTDS versions alongside the pyodbc versions or if this triggers an idea for anyone else. Sorry for the headaches, Mike!
@mkleehammer Hey Mike, I'm creating a Digital Ocean Droplet with a solid repro for you. It'll have a few virtualenvs to show what's going on.
How can I get in touch with you to provide credentials? Want to email me a public key? tim [at] pyphilly [dawt] org
@mkleehammer @FlipperPA Hi Mike,
Our temporary solution was to change the ORM query to use a raw sql call.
Old
configs = WordpressConfig.objects.all()
New
configs = WordpressConfig.objects.raw('SELECT [wordpress_config_id], [url_base], [api_key], [private_key], [form_id], [form_json], cast([form_xml]AS NVARCHAR(MAX)) as form_xml, [entries_json], cast([entries_xml] AS NVARCHAR(MAX)) as entries_xml, [title] FROM[REPORTS].[dbo].[wordpress_api_wordpressconfig]')
Casting the XML to nvarchar got us to a functional application.
Can you try the subinterpreter branch? I'm pretty sure that's the issue. I added use of the UUID class - most of the other classes were implemented in C which are shared among subinterpreters.
I'm on 4.0.16 and I got this problem when trying to connect against a new SQL Server 2012 instance via Freetds from Ubuntu (both 16.04 LTS and and 16.10 zesty). It worked against a different SQL Server instance (we not yet know if there is any difference in the server config; the schemas and the data should be the same). The stacktraces:
*** Error in `/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6': corrupted size vs. prev_size: 0x0000000002853c50 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f6a7871c7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7e9dc)[0x7f6a787239dc]
/lib/x86_64-linux-gnu/libc.so.6(+0x81cde)[0x7f6a78726cde]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xba)[0x7f6a78729dca]
/usr/lib/x86_64-linux-gnu/libodbc.so.2(+0x3ab8c)[0x7f6a6b950b8c]
/usr/lib/x86_64-linux-gnu/libodbc.so.2(+0x693a)[0x7f6a6b91c93a]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(_Z17GetConnectionInfoP7_objectP10Connection+0x2c1)[0x7f6a6bb8ab61]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(_Z14Connection_NewP7_objectbblbS0_R6Object+0x54d)[0x7f6a6bb8c61d]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(+0x924e)[0x7f6a6bb8824e]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(PyCFunction_Call+0xe9)[0x4b1de9]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyEval_EvalFrameDefault+0x693a)[0x549cea]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x542035]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyFunction_FastCallDict+0x129)[0x54ae09]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyObject_FastCallDict+0x1e0)[0x456660]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyObject_Call_Prepend+0xca)[0x45675a]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(PyObject_Call+0x5c)[0x45633c]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyEval_EvalFrameDefault+0x40e6)[0x547496]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x542035]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x5422e7]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyEval_EvalFrameDefault+0x3887)[0x546c37]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x542035]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x5422e7]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyEval_EvalFrameDefault+0x4ad4)[0x547e84]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x542035]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyFunction_FastCallDict+0x1ec)[0x54aecc]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyObject_FastCallDict+0x1e0)[0x456660]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyObject_Call_Prepend+0xca)[0x45675a]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(PyObject_Call+0x5c)[0x45633c]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x4ce80b]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6[0x4c7eda]
/bi/project/testing/venv/91e450c3bdf6d8f8851adeeb7b02519f/bin/python3.6(_PyObject_FastCallDict+0xa2)[0x456522]
[...]
and
*** Error in `/home/js/projects/project/.venv/bin/python3.6': free(): invalid next size (fast): 0x0000564687686da0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f420252108b]
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7f420252ac3a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f420252ed2c]
/usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLDriverConnectW+0x837)[0x7f41e0a04997]
/home/js/projects/project/.venv/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(_Z14Connection_NewP7_objectbblbS0_R6Object+0x2bf)[0x7f41e0c599ff]
/home/js/projects/project/.venv/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(+0xc74e)[0x7f41e0c5174e]
/home/js/projects/project/.venv/bin/python3.6(PyCFunction_Call+0xe9)[0x564685421949]
/home/js/projects/project/.venv/bin/python3.6(_PyEval_EvalFrameDefault+0x759c)[0x5646854bcdec]
/home/js/projects/project/.venv/bin/python3.6(+0x19849d)[0x5646854b449d]
/home/js/projects/project/.venv/bin/python3.6(_PyFunction_FastCallDict+0x137)[0x5646854bd447]
/home/js/projects/project/.venv/bin/python3.6(_PyObject_FastCallDict+0x1d0)[0x5646853b8c60]
/home/js/projects/project/.venv/bin/python3.6(_PyObject_Call_Prepend+0xcc)[0x5646853b8d5c]
/home/js/projects/project/.venv/bin/python3.6(PyObject_Call+0x3a)[0x5646853b893a]
/home/js/projects/project/.venv/bin/python3.6(_PyEval_EvalFrameDefault+0x2e9f)[0x5646854b86ef]
/home/js/projects/project/.venv/bin/python3.6(+0x19849d)[0x5646854b449d]
/home/js/projects/project/.venv/bin/python3.6(+0x198764)[0x5646854b4764]
/home/js/projects/project/.venv/bin/python3.6(_PyEval_EvalFrameDefault+0x47a3)[0x5646854b9ff3]
/home/js/projects/project/.venv/bin/python3.6(+0x19849d)[0x5646854b449d]
/home/js/projects/project/.venv/bin/python3.6(+0x198764)[0x5646854b4764]
/home/js/projects/project/.venv/bin/python3.6(_PyEval_EvalFrameDefault+0x45ea)[0x5646854b9e3a]
/home/js/projects/project/.venv/bin/python3.6(+0x19849d)[0x5646854b449d]
/home/js/projects/project/.venv/bin/python3.6(_PyFunction_FastCallDict+0x1ff)[0x5646854bd50f]
/home/js/projects/project/.venv/bin/python3.6(_PyObject_FastCallDict+0x1d0)[0x5646853b8c60]
/home/js/projects/project/.venv/bin/python3.6(_PyObject_Call_Prepend+0xcc)[0x5646853b8d5c]
/home/js/projects/project/.venv/bin/python3.6(PyObject_Call+0x3a)[0x5646853b893a]
...
What version of driver manager (unixODBC) are you using? You can run the command odbcinst -j
to obtain the version. The latest is 2.3.4 .
Providing an ODBC trace would also be helpful. You can enable tracing by setting the Trace and TraceFile options in the [ODBC] section of odbcinst.ini
, e.g.
[ODBC]
Trace=Yes
TraceFile=odbc.log
Then run your application and a tracefile will be generated. Remember to turn off tracing once you are finished.
I tried to run the trace by somehow the file didn't show up (I used an absolute filename). I then updated all python packages to the latest version and recompiled all pip wheels (deleted the pip cache) at now it seems to work.[update] it still crashes about 2 in 3 runs. [/]
I now have the funny result that python crashed after all SQL Server queries are through (and even a bit more, which do not go through pyodbc but through psycopg2) and the python interpreter is shutting down:
*** Error in `/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6': free(): invalid next size (normal): 0x0000000001a989b0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f40643a37e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f40643ac37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f40643b053c]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x46f64b]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x4c2d7d]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x46f6d1]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x4893f7]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x488f2f]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x49ee87]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x4c9ac4]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x43feba]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(_PyGC_CollectNoFail+0x31)[0x440bb1]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(PyImport_Cleanup+0x2ff)[0x56607f]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x425c16]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(Py_Exit+0x1c)[0x4268bc]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6[0x429f61]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(PyErr_PrintEx+0x17d)[0x42a2dd]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(PyRun_SimpleFileExFlags+0x41f)[0x42b0cf]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(Py_Main+0xd85)[0x43f0e5]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(main+0x162)[0x4213a2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f406434c830]
/bi/pricing/production/venv/200111c2452653f1a1131ed86687759b/bin/python3.6(_start+0x29)[0x421479]
I've now updates to unixODBC 2.3.4
and the problem still happens:
Either straight when loading the query:
*** Error in `/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6': corrupted size vs. prev_size: 0x0000000002b8cce0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f64ad49f7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7e9dc)[0x7f64ad4a69dc]
/lib/x86_64-linux-gnu/libc.so.6(+0x81cde)[0x7f64ad4a9cde]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xba)[0x7f64ad4acdca]
/usr/lib/x86_64-linux-gnu/libodbc.so.2(+0x3b9fc)[0x7f648e8e89fc]
/usr/lib/x86_64-linux-gnu/libodbc.so.2(+0x695a)[0x7f648e8b395a]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(_Z17GetConnectionInfoP7_objectP10Connection+0x2c1)[0x7f6492c86b11]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(_Z14Connection_NewP7_objectbblbS0_R6Object+0x54d)[0x7f6492c885ad]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/lib/python3.6/site-packages/pyodbc.cpython-36m-x86_64-linux-gnu.so(+0x91ee)[0x7f6492c841ee]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(PyCFunction_Call+0xe9)[0x4b2a19]
or this when shutting down the interpreter:
*** Error in `/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6': free(): invalid next size (normal): 0x00000000024befa0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe2dce877e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fe2dce9037a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fe2dce9453c]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x4895a7]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x49efe7]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x4c9c14]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x43ffda]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(_PyGC_CollectNoFail+0x31)[0x440cd1]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(PyImport_Cleanup+0x106)[0x565dc6]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x425c46]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(Py_Exit+0x1c)[0x4268ec]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6[0x42a081]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(PyErr_PrintEx+0x17d)[0x42a3fd]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(PyRun_SimpleFileExFlags+0x41f)[0x42b1ef]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(Py_Main+0xd85)[0x43f205]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(main+0x162)[0x4213d2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe2dce30830]
/bi/pricing/production/venv/fd826216a26b814f01e5e94db7819509/bin/python3.6(_start+0x29)[0x4214a9]
I've now switched from freetds to the ms odbc driver (and I needed to specify the port of that named instance) and I don't have crashes anymore (after two tries).
It seems you have isolated the cause of the crash to the FreeTDS driver and not pyODBC, but if it happens again with msodbcsql please do let us know.
I had just deployed v4.0.0 in my application (after a nightmare using pymssql). However, on a subsequent deploy pip reported that v4.0.0 was no longer available. I updated the requirements to v4.0.1, but now as soon as my application tries to write to the DB, the worker process segfaults. I copied the virtual environment back from a server which was running v4.0.0, and all is well again. Can you please make v4.0.0 available on PyPI again?