search

twosigma / flint

A Time Series Library for Apache Spark
Apache License 2.0
993 stars 184 forks source link

After summarizeWindows flint_df can't able to convert into spark df #62

Open Ashokcs94 opened 5 years ago

Ashokcs94 commented 5 years ago

I can able to convert spark df to flint df and apply the summarizeWindows function also working. After applying any flint function , I can't able to access the df (like i can't convert back to spark df, or can't able to save the result file.

Even flint.show() or spark _df.show() getting error. But function is working correctly, i cant able to access the result df.

icexelloss commented 5 years ago

Hi,

Flint DF is also a Spark DF, you should be able to call regular Spark functions on it. On Tue, Feb 12, 2019 at 2:13 AM Ashokcs94 notifications@github.com wrote:

I can able to convert spark df to flint df and apply the summarizeWindows function also working. After applying any flint function , I can't able to access the df (like i can't convert back to spark df, or can't able to save the result file.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/twosigma/flint/issues/62, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwbrIyU5xPZNcAwb5Cx6jCglXL4d9tOks5vMmmzgaJpZM4a1v6L .

Ashokcs94 commented 5 years ago

But if calls regular spark function on top of flint DF is not working ? I using (Pyspark) Spark version 2.3 and install the ts-flint (using pip in python3) in spark submit i have added the jars and pyFiles (as flint-0.6.0.jar file). I can able to convert the spark df to flint df (by using flint_df = flintContext.read.dataframe(s_df) ). after I am applied flint function (summarizeWindows) i got the new flint df. With the new flint df i can't able to apply any spark df function (eg.. even in new flint sd flint_df.show() is throwing error.)

Is my Jar issue ? or i missed any installation else ?

I didn't get proper docs for the installation and setup of ts-flint, could you please provide the proper doc in urgent manner.

icexelloss commented 5 years ago

Can you show what exception are you getting On Wed, Feb 13, 2019 at 8:39 AM Ashokcs94 notifications@github.com wrote:

But if calls regular spark function on top of flint DF is not working ? I using (Pyspark) Spark version 2.3 and install the ts-flint (using pip in python3) in spark submit i have added the jars and pyFiles (as flint-0.6.0.jar file). I can able to convert the spark df to flint df (by using flint_df = flintContext.read.dataframe(s_df) ). after I am applied flint function (summarizeWindows) i got the new flint df. With the new flint df i can't able to apply any spark df function (eg.. even in new flint sd flint_df.show() is throwing error.)

Is my Jar issue ? or anything else ?

Please help me out.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/twosigma/flint/issues/62#issuecomment-463202298, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwbrMITMGUkNQ9F3_p0wKxXbbxu__-eks5vNBV-gaJpZM4a1v6L .

Ashokcs94 commented 5 years ago

save_to_parquet_error.txt.txt flint_df_show_error.txt.txt

I have attached 2 type of error i am getting. (1 - flint_df.write.parquet error , 2 - flint_df.show() error).

icexelloss commented 5 years ago

Looks like your executors don't have flint jar in the class path