search

astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
https://astronomer-providers.rtfd.io/
Apache License 2.0
128 stars 25 forks source link

Discuss & prepare plan for deprecating astronomer-providers #1404

Closed pankajkoti closed 6 months ago

pankajkoti commented 6 months ago

We can add items to above action items list post our meeting.

pankajkoti commented 6 months ago

The plan could be tentatively as below:

phanikumv commented 6 months ago

Please refer to https://github.com/astronomer/astronomer-providers/issues/1377

phanikumv commented 6 months ago

Also add the points in https://www.notion.so/astronomerio/Draft-Reduce-maintenance-effort-and-keep-backward-Compatibility-for-astronomer-providers-after-con-e16857c5ea814c71afe5009c674d0d67

The plan could be tentatively as below:

  • Merge deprecation PRs to a deprecation branch and keep this branch in sync with main
  • Bump minimum versions of OSS providers to latest or the min version that contain deferrable support for all operators
  • Have a single 2.0.0 release for all the deprecated providers
Lee-W commented 6 months ago

I just finished browsing through all the operators we have and finished this list https://github.com/astronomer/astronomer-providers/issues/1377#issuecomment-1864340191. I will update my future process on this list as well. Some of the operators have not yet contributed back. Most of the minor difference relates to checking before deferring. Regarding GKEStartPodOperatorAsync, OSS airflow deprecates our original implementation and I think we should follow.

pankajkoti commented 6 months ago

Meeting notes (2nd Jan, 2024 - 5.30 PM IST)

Additional points to consider and include in Notion draft

  1. Current released version for astronomer-providers is 1.18.4 as on 1st Jan, 2023. We should create a v-1-18-stable branch from the current main.

  2. Once the v-1-18-stable branch has been created, we can merge deprecation PRs to main.

  3. The deprecation PRs should update minimum version of the depending upstream provider to the latest version of the provider or to at least a version that has the support for the deferrable param.

  4. Release approaches for deprecation PRs:

    1. Release all deprecations in one go in a 1.19.0 release. This will mostly be included in RT 10.x release. We could potentially need a longer time to get all deprecations in place before making a release. The advantage here is easier maintenance wrt to releasing, debugging, bug fixing and runtime image maintenance.
    2. We can also release in chunks e.g. 1.19 for Amazon, 1.20 for Azure etc. but that would mean we need to keep creating more v-19-stable branch , v-1-20-stable branches and leaves a wider room for back-porting fixes. The advantage here is that we can do quick releases provider-by-provider

    The team prefers to go with approach a (release all in one go).

  5. For, 1.19, increase the minimum Airflow version to 2.7 in setup.cfg. This is also the Airflow version in RT 10 and also Airflow OSS providers follow a similar fashion where they keep on bumping minimum Airflow version.

  6. Prepare QA plan which includes testing scenarios like rollback of runtime image. Check customer usage dashboard for crucial operators and ensure good QA for these operators post deprecation.

  7. Highlight 1.19 related notes in public documentation for astronomer-providers

Side notes (TO-DOs)

  1. Maintain a list of OSS provider PRs that are being worked upon in a table on the Notion doc that are needed for the deprecation of operator and that the provider needs to wait for upstream OSS provider to be released
  2. Mention the PR ETAs and release ETAs in the plan for each of the providers

Questions to seek answers for

  1. How long should we keep supporting 1.18.x?
  2. Does the approach for creating v-1-18-stable sound good? Any bug that is reported for versions < 1.19, would necessitate a patch release for 1.18.x (which has to be based upon v-1-18-stable branch and not main branch). Check the approach with @kaxil and seek review
  3. Meet CRE team to understand how customers upgrade and check if the above points seem reasonable. @pankajkoti is planning to meet @abhishekbhakat on 3rd Jan, 2024 together with @vatsrahul1001
pankajkoti commented 6 months ago

Overall the approach looks fine to @abhishekbhakat . Abhishek is going to share the Notion doc with the CRE team and discuss with them the plan.

Some questions from Abhishek:

  1. What will happen with the astronomer-providers package in the long term once deprecation release happens?
  2. For customers using older runtime image (& hence lower Airflow version), is there a way to restrict users installing astronomer-providers to install a version < 1.19. Some customers have an unpinned version of astronomer-providers in their requirements.txt. Test here that for a runtime image, if user has apache-airflow=2.3, and they try to install astronomer-providers unpinned, it should install a version < 1.19 and does not upgrade airflow version.
phanikumv commented 6 months ago

@pankajkoti Lets not share draft document to CRE. It should implement all the suggestions wrt planning, get reviewed by our team first, and post sign off we will share with CRE

pankajkoti commented 6 months ago

okay @phanikumv

vatsrahul1001 commented 6 months ago

Notion docs is done, waiting for final reviews

pankajkoti commented 6 months ago

waiting for another review of the Notion doc, suggestions till date have been incorporated. @phanikumv might be able to review by today EOD.

pankajkoti commented 6 months ago

The review has received good comments and those are incorporated. Since the plan is ready and team discussions have taken place, I am closing this issue. We can continue following up conversation if any on the Notion doc.

cc: @phanikumv