JovanVeljanoski
commented
4 years ago Hi @anubhav-nd wor I do not see anything wrong in your code example, in fact that should work. I have not been able to reproduce the issue locally: this code example works just find (10M rows), you can try it out yourself:
import vaex
import vaex.ml
import hashlib
import numpy as np
def get_hash(x):
return hashlib.sha256(x.encode('utf-8')).hexdigest()
# Create a mock dataset
df_titanic = vaex.ml.datasets.load_titanic()
names = np.random.choice(df_titanic['name'].values, size=10_000_000)
df_mock = vaex.from_arrays(names=names)
# Hashing
df_mock['hash'] = df_mock.apply(f=get_hash, arguments=['names'])
df_mock = df_mock.materialize('hash')
# Export
df_mock.export_arrow('/Users/jovan/Desktop/mock.arrow')If the above example works for you (the file created will be just under 1GB), that leads me to believe it is something to do with the arrow file itself. Looking at the error message, do you perhaps do some filtering, selections or slicing? The error message is very similar to doing this:
import vaex
import vaex.ml
df_titanic = vaex.ml.datasets.load_titanic()
df_filter = df_titanic[df_titanic.age > 45]
print(len(df_filter)) # The result here is 155
my_index = np.arange(155)
df_filter['my_index'] = my_indexwhich raises:
ValueError: Array is of length 155, while the length of the DataFrame is 155 due to the filtering, the (unfiltered) length is 1309.Hope this helps.
anubhav-nd
maartenbreddels
xinnyuann
crliu95
Hi,
I have a dataset with ~1 million rows and I want to run apply function on a column (not the best thing to do, but let's say we just want to do it). I am doing this the following way:
when I try to access
data.hashon the new file, I get the following error:Error evaluating: ValueError('array is of length 262144, while the length of the DataFrame is 1179500')The same steps work fine on datasets with much lesser no. of rows say ~100k.
The only reference to this problem that I could find was at https://github.com/vaexio/vaex/issues/355 but there it has been said that this problem is for datasets larger than memory of machine which I think will not be the situation in my case.
Am I doing something wrong here?