ing-bank / popmon

Monitor the stability of a Pandas or Spark dataframe ⚙︎
https://popmon.readthedocs.io/
MIT License
493 stars 33 forks source link

Error with pydantic when using some custom settings in the report generation #239

Closed gus-morales closed 2 years ago

gus-morales commented 2 years ago

With version 1.0.0, when using custom settings in df.pm_stability_report() like show_stats, I get an error stating such option is not allowed:

ValidationError: 2 validation errors for Settings

I couldn't reproduce it when using popmon==0.9.0.

mbaak commented 2 years ago

Thanks for reporting! Will try to reproduce this.

sbrugman commented 2 years ago

@gus-morales The syntax for configuration has changed (the version increase indicates a breaking change). Are you using the new syntax? The validation errors should actually be a feature, not a bug.

The docs provide examples and descriptions of how to configure popmon. The full range of options can be found in the config.py.

Specifically for show_stats the code would look somewhat like this:

from popmon.config import Settings

settings = Settings()
settings.report.show_stats = ["distinct*",
        "filled*",
        "nan*",
        "mean*",
]

df.pm_stability_report(settings=settings)
sbrugman commented 2 years ago

In order to clarify this in the future we have updated the documentation: https://popmon.readthedocs.io/en/latest/configuration.html#using-settings-for-configuration https://popmon.readthedocs.io/en/latest/popmon.html#module-popmon.config