search

ydataai / ydata-profiling

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
https://docs.profiling.ydata.ai
MIT License
12.53k stars 1.69k forks source link

Feature Request #1536

Open liyaskerj opened 9 months ago

liyaskerj commented 9 months ago

Missing functionality

As a frequent user of ydata_profiling am encountering the below issue

In the given dataset for the numeric datatype column we have to exclude empty cell while calculating 'sum'. When we are not including empty cells then the value of sum is coming as 'NaN'. From our side if we are replacing empty with '0' then it is impacting 'min' value, if we are replacing with some other value then it is impacting the data type of the corresponding column.

Proposed feature

we have to exclude empty or null cell of the numeric data type columns while calculating 'sum'. When we are not including empty cells then the value of sum is coming as 'NaN'

Alternatives considered

Below logic in describe_numeric_spark.py is the place where 'sum' is been is calculated. Please correct me if am wrong

@describe_numeric_1d.register def describe_numeric_1d_spark( config: Settings, df: DataFrame, summary: dict ) -> Tuple[Settings, DataFrame, dict]: """Describe a boolean series.

Args:
    series: The Series to describe.
    summary: The dict containing the series description so far.

Returns:
    A dict containing calculated series description values.
"""

stats = numeric_stats_spark(df, summary)
summary["min"] = stats["min"]
summary["max"] = stats["max"]
summary["mean"] = stats["mean"]
summary["std"] = stats["std"]
summary["variance"] = stats["variance"]
summary["skewness"] = stats["skewness"]
summary["kurtosis"] = stats["kurtosis"]
summary["sum"] = stats["sum"]

Additional context

We are building wheel file from our code and installing the same in databricks cluster and trying to do exploratory data analysis of the given source dataset in CSV file format

liyaskerj commented 8 months ago

@azory-ydata @gliptak @akx @mattf Please let me know is it possible to implement this feature from your end. Thank You