[FEATURE] Add an option to make `expect_column_pair_values_A_to_be_greater_than_B` work at date resolution for datetime column comparisons.

great-expectations / great_expectations

Always know what to expect from your data.

https://docs.greatexpectations.io/

Apache License 2.0

9.99k stars 1.54k forks source link

[FEATURE] Add an option to make `expect_column_pair_values_A_to_be_greater_than_B` work at date resolution for datetime column comparisons. #3418

Closed bentocg closed 3 years ago

bentocg commented 3 years ago

There are cases where we want to compare a column that has detailed datetime info with one that doesn't. In these cases, the 'H:M:S' part is often filled with zeroes, making the column that does not have detailed info more recent than the column that does.

It would be useful to have an option to only check across dates in these cases, maybe a boolean "datetime_as_date" argument that defaults to False.

NathanFarmer commented 3 years ago

Hi @bentocg! Thank you for raising this issue. We will review internally and respond soon.

NathanFarmer commented 3 years ago

Hi @bentocg. We get requests for similar features from time to time, but internally we have decided that it is not something we want to include in the library. The recommendation is to either move these types of transformations into the ETL processes of your data pipelines (as opposed to the data quality processes), or create a new Expectation that performs this task every time (without the use of an argument flag). Additional communication is forthcoming on this subject for the rest of the community's benefit.