search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
713 stars 134 forks source link

upgrade to 3.4 and fix interface changes #178

Closed anqini closed 8 months ago

anqini commented 9 months ago

Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked.

Issue #, if available:

https://github.com/awslabs/python-deequ/issues/148

Forward compatibility issue for scala deequ version >=2.0.4.

Description of changes:

Update the invocation code according to the change of the Scala "deequ-2.0.6-spark-3.4" version interfaces

The interface changes include

Logic change

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

chenliu0831 commented 9 months ago

Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked.

Currently we don't want to maintain separate branch for different version combinations. I agree we shouldn't make the code fractional to achieve the goal and we are working with a long-term plan with changes in Deequ core ideally. I'll share more details around early Jan.

Let's keep this PR open for more ideas.

anqini commented 9 months ago

Thanks for your update. That sounds good.

发件人: Chenyang Liu @.> 发送时间: Thursday, December 14, 2023 11:18:31 PM 收件人: awslabs/python-deequ @.> 抄送: Angelo Ni @.>; Author @.> 主题: Re: [awslabs/python-deequ] upgrade to 3.4 and fix interface changes (PR #178)

Note: Can we create a separate branch for this change to merge into? The changes are only compatible with scala-deequ version >=2.0.4 and not backward compatible. It will look fractional if we make it work universally since for the same class both new and old interfaces need to be invoked.

Currently we don't want to maintain separate branch for different version combinations. I agree we shouldn't make the code fractional to achieve the goal and we are working with a long-term plan with changes in Deequ core ideally. I'll share more details around early Jan.

Let's keep this PR open for more ideas.

― Reply to this email directly, view it on GitHubhttps://github.com/awslabs/python-deequ/pull/178#issuecomment-1856041367, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK7M4FCIMNLP76WRBDXMIJDYJMKEPAVCNFSM6AAAAABAUDCBQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGA2DCMZWG4. You are receiving this because you authored the thread.Message ID: @.***>

LeandroLTM commented 8 months ago

Hey @anqini @chenliu0831, is there a potential date to merge this PR? Looks like Deequ is already adding support to Spark 3.5

chenliu0831 commented 8 months ago

Closing this PR for now - we do not plan to maintain multiple branches and the changes mostly likely have to go to Deequ

theopilbeam commented 7 months ago

@chenliu0831 could you explain more the backwards compatibility requirements for pydeequ? Is it strictly required that all future pydeequ versions continue to support all of the currently supported spark & deequ versions, or is there scope for dropping support?