Blosc / bcolz

A columnar data container that can be compressed.
http://bcolz.blosc.org
959 stars 149 forks source link

fromiter - unable to specify names #357

Open de-code opened 7 years ago

de-code commented 7 years ago

It appears it isn't possible to specify with the names when using fromiter:

Based on the tutorial:

N = 10
bcolz.fromiter(((i,i*i) for i in xrange(N)), dtype="i4,f8", count=N, names=['a', 'b'])

Expected: ctable with the columns 'a' and 'b'

Instead: Exception claiming that the lengths don't match. That is because the code in fromiter is creating an empty ctable where it passes through the keyword arguments (including names) and then adds columns. That means at the creation time len(columns) is zero.

jdavid commented 6 years ago

not tried, but I think you can pass the names in the dtype:

N = 10
dtype = [('a', 'i4'), ('b', 'f8')]
bcolz.fromiter(((i,i*i) for i in xrange(N)), dtype=dtype, count=N)
ckingdev commented 6 years ago

Including the names in the dtype as jdavid suggests does work.

This has very poor documentation, though. From readthedocs:

kwargs : list of parameters or dictionary

Any parameter supported by the carray/ctable constructors.

And the result:

ValueError: columns and names must have the same length

Which gives no information about columns or names, nor is there any way to find out what the problem is besides looking at the source. It would save a lot of headaches to have more informative error messages.