search

JuliaDatabases / ODBC.jl

An ODBC interface for the Julia programming language
https://odbc.juliadatabases.org/stable
Other
106 stars 63 forks source link

OutOfMemory when selecting VARCHAR with the Snowflake ODBC driver #260

Closed akdor1154 closed 4 years ago

akdor1154 commented 4 years ago

Snowflake is very fond of reporting its varchar(max) size of 16MB as the size of text columns in its result sets. As you can imagine this causes OOMs very quickly as ODBC.jl tries to allocate (16MB × nrows). A quick hack of 

const MAX_CSIZE = 1024 # arbitrary
...
        if csizes[x] == 0 || csizes[x] > MAX_CSIZE
            longtexts[x] = ctypes[x] in (API.SQL_VARCHAR, API.SQL_WVARCHAR, API.SQL_VARBINARY)
        end

in Query.jl seems to avoid the problem. I'm happy to submit a PR containing this, but I've never looked at the ODBC spec before and I'm not even a C programmer so I'm a bit out of my depth in expertise here. Is this an acceptable fix? If not, is there any other method?

quinnj commented 4 years ago

I'm digging in to a solution for this; sorry for the slow reply.

quinnj commented 4 years ago

Ok, merged solutions for this in https://github.com/JuliaDatabases/ODBC.jl/pull/263. You can do DBInterface.execute(conn, query; iterate_rows=true) in order to force data-fetching to only buffer one row at a time when fetching results. Also by default, any query that has columns with a size over 4MB will by default only iterate rows.