great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.86k stars 1.52k forks source link

Unable to convert Checkpoint results to serialized json format #10367

Closed PDebasish closed 2 weeks ago

PDebasish commented 2 weeks ago

Describe the bug I am trying to convert the results obtained from checkpoint to a json normalized format and read into a pyspark dataframe. Appreciate your help on the same. Environment Set up.txt

To Reproduce I have included the code in the attachment.

Expected behavior A normalized json format which can be read to a pyspark dataframe.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

PDebasish commented 2 weeks ago

I am trying to convert the checkpoint results to serialized json format using the below code:

Code: df=checkpoint_results.to_json_dict() display(df)

Error: AttributeError: 'CheckpointResult' object has no attribute 'to_json_dict'

adeola-ak commented 2 weeks ago

hi, this behavior is expected. the to_json_dict() attribute does not exist on CheckpointResult however it does look like it exists on ExpectationValidationResult

PDebasish commented 2 weeks ago

Thanks @adeola-ak for the response. Can you please advise how can I de-serialize the checkpoint results and read it in a tabular format in a pyspark dataframe. As we plan to store it in a delta table in lakehouse.

adeola-ak commented 2 weeks ago

The CheckpointResult has a describe() method that returns a JSON string description of the result. You can convert this as needed. Beyond this, Great Expectations doesn't offer any built-in solutions, so you would need to implement a custom approach.

adeola-ak commented 2 weeks ago

I will be closing this issue - I hope the above advice helps. Thank you!