Turning off include_unexpected_rows is ignored for "COMPLETE" result format

[ezgigurkan]

The bug: I've been trying to get PySpark queries (unexpected_index_query) for validations that fail wtihout including all those failed rows in the results since there can be a very large number of rows.

It doesn't seem possible to suppress unexpected_list and unexpected_index_list values in the "COMPLETE" result format.

To Reproduce: Here is the format I'm using, both at expectation-level and checkpoint-level:

"result_format": {
    "result_format": "COMPLETE",
    "include_unexpected_rows": False,
    "partial_unexpected_count": 0,
    "return_unexpected_index_query": True
}

Expected behavior: I would expect no results for unexpected_list or unexpected_index_list if include_unexpected_rows is set to False.

Environment:

• Great Expectations Version: 0.17.6
• Datasource: Databricks/Spark
• Cloud environment: Azure Databricks Notebook

[HaebichanGX]

Hi @ezgigurkan thank you for reaching out to us on this issue. Let me put it into backlog for our review.

[HaebichanGX]

Hi @ezgigurkan so we took an internal review, and unexpected_list populating is actually a feature, not a bug. The "include_unexpected_rows": False attribute is a knob that is intended to dramatically reduce every information on the rows that failed to just the value that it failed on. It's not intended to not show any results in the unexpected_list.

To not show the list itself, you can message around with the result_format parameter. Here is the reference doc: https://docs.greatexpectations.io/docs/reference/expectations/result_format

[molliemarie]

Hello @ezgigurkan. With the upcoming launch of Great Expectations Core (GX 1.0), we are closing old issues posted regarding previous versions. Moving forward, we will focus our resources on supporting and improving GX Core (version 1.0 and beyond). If you find that an issue you previously reported still exists in GX Core, we encourage you to resubmit it against the new version. With more resources dedicated to community support, we aim to tackle new issues swiftly. For specific details on what is GX-supported vs community-supported, you can reference our integration and support policy.

To get started on your transition to GX Core, check out the GX Core quickstart (click “Full example code” tab to see a code example).

You can also join our upcoming community meeting on August 28th at 9am PT (noon ET / 4pm UTC) for a comprehensive rundown of everything GX Core, plus Q&A as time permits. Go to https://greatexpectations.io/meetup and click “follow calendar” to follow the GX community calendar.

Thank you for being part of the GX community and thank you for submitting this issue. We're excited about this new chapter and look forward to your feedback on GX Core. 🤗