vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.25k stars 590 forks source link

Not working with tensorflow #2166

Closed JuliaJ99 closed 2 years ago

JuliaJ99 commented 2 years ago

This is the error I got: image

when I called the fit method while I was trying to run a neural network model using tensorflow. I largely followed the docs example here except I didn't scale my data, added more layers, changed activation functions, and used different optimizer, metrics for compiling. https://vaex.readthedocs.io/en/latest/tutorial_ml.html#Keras-example

and it shouldn't hopefully be a package issue as vaex-ml is there image

Any idea?

Software information

JovanVeljanoski commented 2 years ago

Can you share your code..?

From where error it looks like you are using a dataframe somewhere where a string or an expression is exptected

JuliaJ99 commented 2 years ago

https://nbviewer.org/github/JuliaJ99/Kaggle-projects/blob/ad5500762054ea650ff917a6cef0f4d15aaf7851/NB1.ipynb

If that's easier to read or I just posted it on github https://github.com/JuliaJ99/Kaggle-projects/blob/ad5500762054ea650ff917a6cef0f4d15aaf7851/NB1.ipynb

JovanVeljanoski commented 2 years ago

So as the error says... in your case target appears to be a dataframe. It should be an expression, one that which exists in the training (and validation) data

JuliaJ99 commented 2 years ago

I am a bit confused, what do you mean by expression? I had added the target by means of a join, that can't work?

I did check and yes the target exists in both training and validation sets image

JovanVeljanoski commented 2 years ago

You use the variable target in

gen_train = df_train.ml.tensorflow.to_keras_generator(features=cols_use, target=target, batch_size=512)

while you should use the column or expression name:

gen_train = df_train.ml.tensorflow.to_keras_generator(features=cols_use, target='target' batch_size=512)

if the name of that target column is "target".

Maybe if would help you to go through the tutorial to learn how vaex actually works.

JuliaJ99 commented 2 years ago

I feel so dumb🤦‍♀️, should have named the data frame differently from the vaex column. Thank you so much.

JovanVeljanoski commented 2 years ago

Sure, glad to help!