Reduced calls to 'toPandas'

julioasotodv / spark-df-profiling

Create HTML profiling reports from Apache Spark DataFrames

MIT License

195 stars 77 forks source link

Closed danielenricocahall closed 4 years ago

danielenricocahall commented 4 years ago

Rather than computing both count and distinct count and converting them to Pandas dataframes independently, I consolidated their computation/conversion into one select call.
Edited the categorical 'top' computation to use the top 50 pandas dataframe rather than calling 'toPandas' twice

julioasotodv commented 4 years ago

Merging. Thank you for the optimisation!