search

great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.99k stars 1.54k forks source link

Using allow_cross_type_comparisons in expect_column_pair_Values_A_to_be_greater_than_B raises Not Implemented Error #3373

Closed vivek89007 closed 3 years ago

vivek89007 commented 3 years ago 

    @MetaPandasDataset.column_pair_map_expectation
    def expect_column_pair_values_A_to_be_greater_than_B(
        self,
        column_A,
        column_B,
        or_equal=None,
        parse_strings_as_datetimes=None,
        allow_cross_type_comparisons=None,
        ignore_row_if="both_values_are_missing",
        result_format=None,
        row_condition=None,
        condition_parser=None,
        include_config=True,
        catch_exceptions=None,
        meta=None,
    ):
        # FIXME
        if allow_cross_type_comparisons == True:
            raise NotImplementedError```

``

While using allow_cross_type_comparisons in 0.13.30 version is raising Not Implemented Error...When are we guys planning to add this support? Can someone provide a workaround?
vivek89007 commented 3 years ago

@jcampbell @eugmandel @bhcastleton - Please help

NfnqB commented 3 years ago

Same problem with expect_column_pair_values_to_be_equal. The implementation is simply just throwing NotImplmentedError. Check some others multi-column expectations and seems all having the same problem.

NathanFarmer commented 3 years ago

Hi @vivek89007 and @NfnqB! You you for creating this issue. We are reviewing this internally and we will follow up in the next few days.

NathanFarmer commented 3 years ago

Hi @vivek89007 and @NfnqB! After some internal discussions, we were hoping you could provide a little more context around your specific use cases for this feature. There is a lot of complexity inherent in converting and comparing data types across the various supported backends. Performing those transformations during ETL would seem to be a more natural "line in the sand" between the data pipelines and measuring the quality of the data in those pipelines.

NfnqB commented 3 years ago

@NathanFarmer Mine is with SqlAlchemyDataset and Legacy v2 api Great Expectations version 0.13.31 When trying to run expect_column_pair_values_to_be_equal expectation on debug mode and the function stops at abstract dataset.py with no actual implementation in Sqalchemy_dataset.py, while the other two dataset (pandas_dataset and sparkdf_dataset) have no problem working with it. Maybe it is related to the version not updated enough issue?

vivek89007 commented 3 years ago

I am using v2 LegacyCheckpoint for validation and version 0.13.30 version of great expectations. I am trying to validate a pandas dataset and while using allow_cross_comparisons=True and getting this error as its not implemented.

On Tue, Sep 14, 2021, 7:57 AM NfnqB @.***> wrote:

Mine is with SqlAlchemyDataset and Legacy v2 api Great Expectations version 0.13.31 When trying to run expect_column_pair_values_to_be_equal expectation on debug mode and the function stops at abstract dataset.py with no actual implementation in Sqalchemy_dataset.py, while the other two dataset (pandas_dataset and sparkdf_dataset) have no problem working with it. Maybe it is related to the version not updated enough issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/great-expectations/great_expectations/issues/3373#issuecomment-918741957, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUFQBI4F5OTLGZGNH4552TUB2XI7ANCNFSM5DWMFOOQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

NathanFarmer commented 3 years ago

@NfnqB @vivek89007 I'm sorry I wasn't clear. What I meant by your use case is, how are you intending to use this feature? What two data types are you comparing against each other?

vivek89007 commented 3 years ago

I need to compare strings, floats, int and datetime.

On Tue, Sep 14, 2021 at 6:22 PM Nathan Farmer @.***> wrote:

@NfnqB https://github.com/NfnqB @vivek89007 https://github.com/vivek89007 I'm sorry I wasn't clear. What I meant by your use case is, how are you intending to use this feature? What two data types are you comparing against each other?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/great-expectations/great_expectations/issues/3373#issuecomment-919120123, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUFQBJZHYTXLTHQWELNDGDUB5ARDANCNFSM5DWMFOOQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

NfnqB commented 3 years ago

Im trying to compare two Decimal value from a join query.

NathanFarmer commented 3 years ago

@NfnqB if the data types are the same, why are you needing to use allow_cross_type_comparisons?

vivek89007 commented 3 years ago

Can we expect this fix in the next release?

On Wed, Sep 15, 2021 at 9:21 PM Nathan Farmer @.***> wrote:

@NfnqB https://github.com/NfnqB if the data types are the same, why are you needing to use allow_cross_type_comparisons?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/great-expectations/great_expectations/issues/3373#issuecomment-920143298, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUFQBIY54GK5GBELLCC5P3UCC6JBANCNFSM5DWMFOOQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

NathanFarmer commented 3 years ago

Hi @vivek89007 and @NfnqB. We are intending to deprecate this feature as it has become our position that this should be handled during the ETL process (as opposed to the data quality testing process). More communication around this will be forthcoming, but it would be prudent for you to start looking for alternative ways of performing this transformation.

talagluck commented 3 years ago

I'm going to close this issue, since this feature is being deprecated. Please keep an eye out for a blog post providing more detail in the next couple of weeks!