search

sryza / aas

Code to accompany Advanced Analytics with Spark from O'Reilly Media
Other
1.52k stars 1.03k forks source link

Chapter 3: Convert PySpark DataFrame to Pandas #150

Closed tanthiamhuat closed 2 years ago

tanthiamhuat commented 2 years ago
image image

Is it because recommendations is of type array, it creates some issue when need to convert to Pandas? How do we overcome this?

srowen commented 2 years ago

This is fine, it's a warning that arrow optimizations do not apply to this case, but that's normal.

tanthiamhuat commented 2 years ago

yup, it is a warning.. but I am not sure why I am not allowed to print it out, as that variable 'top_prediction_pandas' is not defined. However, its below code with the variable 'top_prediction_pandas' runs fine. image

srowen commented 2 years ago

There is a typo here - top_prediction_pandas vs top_predictions_pandas

tanthiamhuat commented 2 years ago

oic, thank you, we can close this issue.