django: Reading LONG VARCHAR columns - adapter saves ok, but reads only first character

[GoogleCodeExporter]

I have existing table in db which is declared as LONG VARCHAR (no length
defined, default is 32700).

I mapped this to django TextField with no length models.TextField(). 

But when data is read from the table only first character is retrieved. 
Saving data to this column 
Example:

class Table(models.Model):
    data = models.TextField(_('data'))

>>> Table.objects.create(data="test")
>>> Table.objects.all()[0].data=="t"
True

Saving to db went ok, since in table is really "test" data, but fetching
from table returns only "t".

Original issue reported on code.google.com by trebor74hr@gmail.com on 5 Jan 2010 at 4:31

[GoogleCodeExporter]

I know that LONG VARCHAR are to be deprecated, but I have existing tables with 
it ...
I'll do workaround with migrating them to clob. 

What I additionally found is that this is not the issue of ibm_db_django, but 
ibm_db
(C code - not ibm_db_dbi - i checked in _fix_return_data_type). In example
"DJANGO_SESSION"."SESSION_DATA" is long varchar:

>db2 SELECT "DJANGO_SESSION"."SESSION_DATA" FROM "DJANGO_SESSION" WHERE 1=1
SESSION_DATA
--------------------------------------------------------------------------------
------------
gAJ9cQEoVRJfYXV0aF91c2VyX2JhY2tlbmRxAlUpZGphbmdvLmNvbnRyaWIuYXV0aC5iYWNrZW5k
cy5Nb2RlbEJhY2tlbmRxA1UNX2F1dGhfdXNlcl9pZHEESwF1LmQ3ZjJmM2I5ZTQxNWFmOTc3OWE4
Y2M3ZGE5N2U4YjJm

>>> from django.db import connection
>>> cur = connection.cursor()
>>> cur.execute("""SELECT "DJANGO_SESSION"."SESSION_KEY",
"DJANGO_SESSION"."SESSION_DATA", "DJANGO_SESSION"."EXPIRE_DATE" FROM 
"DJANGO_SESSION"
WHERE 1=1""")
True
>>> # see session data=="g"
>>> cur.fetchall()
[(u'a18edce707c7a1be9a031f1e98708822', 'g', datetime.datetime(2010, 1, 21, 14, 
32, 7,
602000))]
>>> import ibm_db
>>> ibm_db.connect("test", "test", "test")
>>> import ibm_db_dbi
>>> conn = ibm_db_dbi.Connection(con)
>>> cur = conn.cursor()
>>> cur
<ibm_db_dbi.Cursor object at 0x01DF5350>
>>> cur.execute("""SELECT "DJANGO_SESSION"."SESSION_KEY",
"DJANGO_SESSION"."SESSION_DATA", "DJANGO_SESSION"."EXPIRE_DATE" FROM 
"DJANGO_SESSION"
WHERE 1=1""")
True
>>> # again see session data=="g"
>>> cur.fetchall()
[(u'a18edce707c7a1be9a031f1e98708822', 'g', datetime.datetime(2010, 1, 21, 14, 
32, 7,
602000))]

There is something wrong with detecting input_size when fetching from long 
varchar in
ibm_db.pyd (cpython).

Original comment by trebor74hr@gmail.com on 7 Jan 2010 at 1:55

[GoogleCodeExporter]

Thanks Robert for reporting this bug. I have created a patch for this issue. 
you can
find this on attachment. I have done with initial testing against window and 
linux.
But testing is still going on. 
please try this patch and let me know how this works. 

Original comment by rahul.pr...@in.ibm.com on 8 Jan 2010 at 7:12

• Changed state: Fixed
• Added labels: OpSys-All

Attachments:

• ibm_db-1.0_longvarchar_patch.tar.gz

[GoogleCodeExporter]

Thanks for fast response. I work on windows and succeeded to build pyd. Haven't
tested yet, but found several issues in build:

 - I noticed that you copied/moved ibm_db_dbi\ibm_db_dbi.py but in
ibm_db\ibm_db_dbi.py. Hope this is ok.

 - in normal C - you must declare all variables at block start - so I had issues: 
     ibm_db.c(932) : error C2143: syntax error : missing ';' before 'type'
     ibm_db.c(942) : error C2065: 'isNewBuffer' : undeclared identifier
     ibm_db.c(1161) : error C2275: 'PyObject' : illegal use of this type as an expression
     ...
   I corrected this - moved all declarations at block start (code in attachment), but
didn't tested. Don't know if there is a way to make mscpp7.1 compiler accept 
this,
but if not trouble rather use std. C logic.

 - only two warnings are left
     r:\vvs-lib\MsVS.NET_2003\Vc7\bin\cl.exe ... ibm_db.c
     ibm_db.c(4562) : warning C4018: '<' : signed/unsigned mismatch
     ibm_db.c(7093) : warning C4244: 'function' : conversion from 'SQLINTEGER' to
'SQLUSMALLINT', possible loss of data

 - (not related) personally I don't like setuptools, but I needed to install it. Why
you didn't used standard distutils as build/install tool?

 - small notice - os.environ['IBM_DB_DIR'] can be found automatically (e.g. windows
os.environ['ProgramFiles']\IBM\SQLLIB) - and can be overriden by 
os.environ['IBM_DB_DIR']

Original comment by trebor74hr@gmail.com on 8 Jan 2010 at 12:11

Attachments:

• ibm_db.c

[GoogleCodeExporter]

:) It seems to be working:

>>> from django.conf import settings
>>> settings.DATABASE_ENGINE
'ibm_db_django'
>>> from django.contrib.sessions.models import *
>>> Session.objects.all()[0]
<Session: Session object>
>>> s = Session.objects.all()[0]
>>> s.session_data
u'gAJ9cQEoVRJfYXV0aF91c2VyX2JhY2tlbmRxAlUpZGphbmdvLmNvbnRyaWIuYXV0aC5iYWNrZW5k\n
cy5Nb2RlbEJhY2tlbmRx
A1UNX2F1dGhfdXNlcl9pZHEEigEBdS42MWNjODk1ZGMzZmVjZTc1ZDMz\nZjNiOTcyZGI3MTU0Nw==\n
'

Original comment by trebor74hr@gmail.com on 8 Jan 2010 at 12:27

[GoogleCodeExporter]

btw. probably not proper place to place this code, but don't know better: in
attachment you can find small patch to ibm_db_dbi.py that can improve 
performance a
little (if -> elif).

Original comment by trebor74hr@gmail.com on 8 Jan 2010 at 12:33

Attachments:

• ibm_db_dbi-small-perf.diff

[GoogleCodeExporter]

in attachment is py25-win32 build with patch applied (probably working copy 
based on
revision r96).

Original comment by trebor74hr@gmail.com on 9 Jan 2010 at 5:08

Attachments:

• ibm_db-win32-2.5-issue41.zip

[GoogleCodeExporter]

Thanks for pointing out coding defect in our DBI wrapper. Through this changing 
(if
-> elif) we really get a little bit of performance improvement. 

Original comment by rahul.pr...@in.ibm.com on 11 Jan 2010 at 9:23

[GoogleCodeExporter]

Original comment by rahul.pr...@in.ibm.com on 2 Feb 2010 at 5:26

• Changed state: Started

[GoogleCodeExporter]

Fixed in ibm_db-1.0.1

Original comment by rahul.pr...@in.ibm.com on 27 Mar 2010 at 8:44

• Changed state: Fixed