WanderInCode / pyodbc

Automatically exported from code.google.com/p/pyodbc
MIT No Attribution
0 stars 0 forks source link

appcrash when dealing with large number of long varchar fields #380

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
What steps will reproduce the problem?
1. Connect to Filemaker Pro database (I am using Filemaker pro 13, driver 
version 13.02.14 but have experienced this issue across several different 
versions of filemaker pro and filemaker pro drivers)

cnxn = pyodbc.connect(Driver='FileMaker 
ODBC',Server='localhost',DSN='DSN',database='DBASE',UID='UID',PWD='PWD')

2. iterate over a large number of long varchar records. I have tried 
simplifying down to the following where:

len(articles.article) < 40,000 but typically greater than 2k 
and 
the article table has appx 10k records:

cursor.execute("SELECT article FROM articles")
for row in cursor:
    temp = articles.article

I have also tried iterating over the data set in chunks of various sizes. ie. 
SELECT first 20 articles... next 20... etc. iterating over a small SELECT 
usually works better but still crashes. Also using an unordered SELECT usually 
performs better than an ordered select.

What is the expected output? What do you see instead?

I expect to be able to fetch the records without issue or at least throw an 
error but instead always get appcrash. Also app crash does not happen at a 
specific record but randomly. ie. sometimes will crash on the 500th record, 
sometimes on the two-thousandth. Sometimes on an extra long record, sometimes 
on a short record.

What version of the product are you using? On what operating system?
anaconda python 2.7 32bit (have also tried on 64bit and vanilla python)
pyodbc - various versions up to the latest commit as of aug 21, 2014.
various versions of the filemaker odbc driver. currently 13.02.14
Filemaker Pro 13 though have also tried on 11, 12 and FMP Server.

Please provide any additional information below.
I am able to successfully execute this query in Microsoft excel so it seems 
that the error is being caused by pyodbc and not necessarily the filemaker pro 
driver.

Here is a sample ODBC trace where the app crashes. In this instance it crashed 
when processing a short record.

python          570-644 ENTER SQLFetch 
        HSTMT               0x0306E6E0

python          570-644 EXIT  SQLFetch  with return code 0 (SQL_SUCCESS)
        HSTMT               0x0306E6E0

python          570-644 ENTER SQLGetData 
        HSTMT               0x0306E6E0
        UWORD                        1 
        SWORD                        1 <SQL_C_CHAR>
        PTR                 0x0027F7B6 
        SQLLEN                  1024
        SQLLEN *            0x0027F790

python          570-644 EXIT  SQLGetData  with return code 0 (SQL_SUCCESS)
        HSTMT               0x0306E6E0
        UWORD                        1 
        SWORD                        1 <SQL_C_CHAR>
        PTR                 0x0027F7B6 [     559] "...some text..."
        SQLLEN                  1024
        SQLLEN *            0x0027F790 (559)

python          570-644 ENTER SQLFetch 
        HSTMT               0x0306E6E0

Original issue reported on code.google.com by gall.a...@gmail.com on 21 Aug 2014 at 2:16

GoogleCodeExporter commented 8 years ago
error in code above... should read: 

for row in cursor:
    temp = row.article

Original comment by gall.a...@gmail.com on 21 Aug 2014 at 2:18

GoogleCodeExporter commented 8 years ago
update: if I increase the buffer size by changing line 330 of getdata.cpp from 

char tempBuffer[1026]; // Pad with 2 bytes for driver bugs

to

char tempBuffer[96002]; // Pad with 2 bytes for driver bugs

The issue seems to be resolved. Though I cant imagine that setting the buffer 
so large is the desired behavior.

Original comment by gall.a...@gmail.com on 21 Aug 2014 at 6:23