How do I download data to postgresql?

nfrvnikita commented 1 year ago

Hello, everyone!

How can I load the resulting metrics into the postgresql database, in order to further visualize the dashboard in real time with grafana? I took this repository as a reference, but I am using my own data which is uploaded in real time https://github.com/mzadafiya/grafana_evidently_drift_detection , but nothing works.

The error that pops up is:

psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type numeric: ""
LINE 1: ...1'::date,'tech_no','cat','Reference Distribution','',3),('20...

nfrvnikita commented 1 year ago

I think I figured out what the problem was and how to fix it. I need to use ColumnMapping, but another problem arose. Can you tell me what I did wrong?

                    column_mapping = ColumnMapping()
                    column_mapping.categorical_features = df_prev_prod.loc[:, df_prev_prod.dtypes == object].columns
                    column_mapping.numerical_features = list(df_prev_prod.select_dtypes(exclude='object').columns)
                    # data_drift = Dashboard(tabs=[DataDriftTab(verbose_level=False), CatTargetDriftTab(verbose_level=False)])
                    model_profile = Profile(sections=[DataDriftProfileSection()])
                    model_profile.calculate(data_ref[table], df_prev_prod, column_mapping=column_mapping)
                    json_eviden_metrics = json.loads(model_profile.json())
                    evidently_metrics = json_eviden_metrics['data_drift']['data']['metrics']
                    df_metrics = nested_dict_to_dataframe(evidently_metrics)
                    df_metrics.to_sql('data_drift', engine, if_exists='append', index=True)

elenasamuylova commented 1 year ago

Hi @nfrvnikita,

we recently released:

A complete example integration of Evidently with Postgres as a metrics destination: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/postgres_grafana_batch_monitoring
We also simplified JSON/Python dictionary output and introduced parameters to customize JSON output (e.g. define which metrics to include) https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_customize_json_output.ipynb

Both could be helpful with relation to your question.

Could you share if you still face issues with Postgres integration and Column mapping with the latest Evidently version?

evidentlyai / evidently

How do I download data to postgresql? #501