[BUG-REPORT] The right way to add derived column when converting csv to hdf5?

[r3verser]

*Description** Hi, cant get it work! Im converting huge CSV to hdf5 and i need to add new column with value derived from existing one. Error occured TypeError: list indices must be integers or slices, not str in line df["md5"] = [hashlib.md5(val.encode('utf-8')).hexdigest() for val in df['address']] code sample:

import vaex as vs
...
for i, df in enumerate(vs.from_csv('../data.csv', chunk_size=100_000, encoding="ISO-8859-1", usecols=columns, dtype=dtypes)):
    df = df.extract()
    df["md5"] = [hashlib.md5(val.encode('utf-8')).hexdigest() for val in df['address']]
    df.export_hdf5(f'./db_{i:02}.hdf5')

Software information

• Vaex version (import vaex; vaex.__version__):
• {'vaex': '3.0.0', 'vaex-core': '2.0.3', 'vaex-viz': '0.4.0', 'vaex-hdf5': '0.6.0', 'vaex-server': '0.3.1', 'vaex-astro': '0.7.0', 'vaex-jupyter': '0.5.2', 'vaex-ml': '0.9.0', 'vaex-arrow': '0.5.1'}
• Vaex was installed via: pip / conda-forge / from source
• conda
• OS: Win 7 x64

[JovanVeljanoski]

Hi,

Can you please try installing the latest alpha from pip: pip install vaex==4.0.0a13 ?

And try to convert your csv data using this example.

Although I suspect that this like: df["md5"] = [hashlib.md5(val.encode('utf-8')).hexdigest() for val in df['address']] is the problematic one. What does it contain? If you want to store lists, i think you should use the latest alpha and try exporting to an arrow format.

I hope this helps!