Closed shashwatwork closed 5 years ago
Hmm, do you mean performance profiling? You can measure how long evaluating a lazy computation takes (e.g., by surrounding a call to evaluate() with time.time()), but unfortunately there isn't any easy way to see how long each operation in a graph takes (the operations will likely be fused into a single loop, so its hard to tease apart where time should be attributed).
On Fri, Jul 12, 2019 at 12:10 AM SHASHWAT TIWARI notifications@github.com wrote:
Hi,
I would like to perform pandas profiling on particular data frame. Whether it can be achieved through weld.
Thanks
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/weld-project/weld/issues/460?email_source=notifications&email_token=AAKMEYZ7GL5LXKY5LWTJ23LP7AU5TA5CNFSM4ICDKW52YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6ZZCBA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKMEY753A4CODHB66WPFALP7AU5TANCNFSM4ICDKW5Q .
-- Shoumik
No, I'm asking about how to perform data profiling (https://pypi.org/project/pandas-profiling/) on top of DataFrameWeld under grizzly or some other classes.
eg (done on top of pandas DataFrame) p = pandas_profiling.ProfileReport(df,check_correlation=False) p.to_file("profiling01.html")
Here, pandas profiling takes only dataframe as input. Is it possible to give Weld Data frame as input. If yes please let me know how to do.
I had also tried with weld_df.df. basically it will convert weld object to pandas data frame and then work. But time taken is not reduced, it take some what higher than normal profiling.
Thanks
Aha, yes, right now we don't support that feature unfortunately (looking at what it produces, Grizzly and Baloo https://github.com/weld-project/baloo have functions that produce some of these statistics, but not through that interface).
On Mon, Jul 15, 2019 at 3:51 AM SHASHWAT TIWARI notifications@github.com wrote:
No, I'm asking about how to perform data profiling ( https://pypi.org/project/pandas-profiling/) on top of DataFrameWeld under grizzly or some other classes.
eg (done on top of pandas DataFrame)
p = pandas_profiling.ProfileReport(df,check_correlation=False) p.to_file("profiling01.html")
Here, pandas profiling takes only dataframe as input. Is it possible to give Weld Data frame as input. If yes please let me know how to do.
I had also tried with weld_df.df. basically it will convert weld object to pandas data frame and then work. But time taken is not reduced, it take some what higher than normal profiling.
[image: profiling] https://user-images.githubusercontent.com/22785727/61211175-8099d980-a71c-11e9-998d-fe8ae8c514c8.PNG
Thanks
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/weld-project/weld/issues/460?email_source=notifications&email_token=AAKMEY5XYH64TI5QYN3KJ3TP7RJB5A5CNFSM4ICDKW52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5KVTI#issuecomment-511355597, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKMEY765YNEHHSYCOKB3MDP7RJB5ANCNFSM4ICDKW5Q .
-- Shoumik
Thanks cool!
Hi,
I would like to perform pandas profiling on particular data frame. Whether it can be achieved through weld.
Thanks