search

blaze / odo

Data Migration for the Blaze Project
http://odo.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
1.01k stars 138 forks source link

Does odo support more than 255 columns? #569

Open thequackdaddy opened 7 years ago

thequackdaddy commented 7 years ago

Tried to convert a really, really big dataset (45 GB in mssql, around 355 columns and 22 million rows) in through odo and got an error that you can't have more than 255 arguments.

Basically, I was converting the mssql table to a bcolz dataset. I'm not sure where the problem lies.

I'll try to create a toy example later.

llllllllll commented 7 years ago

The python language does not support more than 255 arguments to a single function syntactically, though you can * unpack a large tuple or ** unpack a large dictionary into the function. Were you trying to do something like:

function(
    arg_0,
    arg_1,
    ...,
    arg_300,
)
thequackdaddy commented 7 years ago

Right some pseudocode of what I was doing was this...

import odo

odo.odo(odo.resource('mssql+pyodbc://SQL::Table', schema='dbo'), 'c:/outfile.bcolz', chunksize=100000, expectedlen=22000000)
llllllllll commented 7 years ago

without the traceback it is hard to say what the error is. There are a lot of places this may be.