Status of testing of Apache Airflow 2.7.2rc1

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.7.2rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• [ ] Improve back-reference generation (#33149): @potiuk
• [ ] Better fix for add-back-references (#33207): @potiuk
• [ ] Refactor: Simplify code in breeze (#33273): @eumiro
• [ ] Limit connection resources used in mapped tasks (#33309): @potiuk Linked issues:
  • Flaky test_xcom_map_error_fails_task test (#33178)
• [ ] Refactor: Simplify conditions on len() (#33454): @eumiro
• [ ] Replace strftime with f-strings where nicer (#33455): @eumiro
• [ ] Refactor: Do not create lists we don't need (#33519): @eumiro
• [ ] Bump min-version for docker and docker-compose (#33572): @potiuk @uranusjr Linked issues:
  • Avoid WSL2 ones when finding a context for Breeze (#33547)
• [ ] Use str.splitlines() to split lines (#33592): @eumiro
• [ ] Hide Irrelevant Dag Processor from Cluster Activity Page (#33611): @erdos2n Linked issues:
  • DAG Processor should not be visible in the Cluster Activity Page if there is no stand alone processor (#33497)
• [ ] Refactor: Remove useless str() calls (#33629): @eumiro
• [ ] Refactor: Use random.choices (#33631): @eumiro
• [ ] Refactor: path.rglob("…") does not need "**" (#33669): @eumiro
• [ ] Refactor: Improve detection of duplicates and list sorting (#33675): @eumiro
• [ ] Refactor: Use f-strings (#33734): @eumiro
• [ ] Refactor: tmp_path in tests/dag_processing (#33740): @eumiro
• [ ] Refactor: Use inplace .sort() (#33743): @eumiro
• [ ] Refactor: Replace lambdas with comprehensions (#33745): @eumiro
• [ ] Refactor: remove unused state - SHUTDOWN (#33746): @Bisk1
• [x] E731: replace lambda by a def method in Airflow core (#33758): @hussein-awala
• [x] Replace type func by isinstance in fab_security manager (#33760): @hussein-awala
• [x] Use literal dict instead of calling dict() in Airflow core (#33762): @hussein-awala
• [x] remove unnecessary map and rewrite it using list in Airflow core (#33764): @hussein-awala
• [x] Replace single quotes by double quotes in all Airflow modules (#33766): @hussein-awala
• [x] Merge multiple isinstance calls for the same object in a single call (#33767): @hussein-awala
• [x] Use a single statement with multiple contexts instead of nested statements in core (#33769): @hussein-awala
• [ ] Improve importing the modules in Airflow www package (#33810): @hussein-awala
• [ ] Refactor unneeded 'continue' jumps in utils (#33836): @eumiro
• [ ] Refactor unneeded 'continue' jumps in dag processing (#33839): @eumiro
• [ ] Refactor unneeded 'continue' jumps in api (#33842): @eumiro
• [ ] Refactor unneeded 'continue' jumps in configuration (#33844): @eumiro
• [ ] Refactor unneeded 'continue' jumps in cli (#33845): @eumiro
• [ ] Refactor unneeded 'continue' jumps in jobs (#33846): @eumiro
• [ ] Refactor unneeded 'continue' jumps in setup.py (#33848): @eumiro
• [ ] Refactor unneeded 'continue' jumps around the repo (#33849): @eumiro
• [x] replace unnecessary dict comprehension by dict() in core (#33858): @hussein-awala
• [x] Cleanup: remove remnants of 'shutdown' task instance state (#33893): @Bisk1 Linked issues:
  • Refactor: remove unused state - SHUTDOWN (#33746)
• [x] Replace sequence concatination by unpacking in Airflow core (#33934): @hussein-awala
• [x] Replace single element slice by next() in Airflow tests (#33938): @hussein-awala
• [x] Replace dict.items by dict.values when key is not used in Airflow core (#33940): @hussein-awala
• [x] Avoid creating unnecessary list when parsing stats datadog tags (#33943): @hussein-awala
• [x] Add cancel_trigger_ids to to_cancel dequeue in batch (#33944): @hussein-awala
• [x] Remove useless string join from core (#33969): @hussein-awala
• [x] Move the try outside the loop when this is possible in Airflow core (#33975): @hussein-awala
• [x] Replace == by is for types comparison (#33983): @hussein-awala
• [x] replace loop by any when looking for a positive value in core (#33985): @hussein-awala
• [x] Combine similar if logics in core (#33988): @hussein-awala
• [x] Restore rendering of provider configuration with version_addded (#34011): @potiuk @Taragolis Linked issues:
  • version_added field in configuration option doesn't work correctly in providers documentation (#34005)
• [x] Remove [project] section from pyproject.toml (#34014): @risicle @Taragolis Linked issues:
  • pyproject.toml [project] section without name and version attributes is not pep 621 compliant (#33854)
• [ ] Remove infinite animation for pinwheel, spin for 1.5s (#34020): @pankajkoti @mj-dd Linked issues:
  • please disable pinwheel animation as it violates ada guidelines (#34019)
• [x] Fix scheduler logic to plan new dag runs by ignoring manual runs (#34027): @Bisk1
• [ ] Consolidate importing of os.path.* (#34060): @eumiro
• [ ] Bring back SHUTDOWN-related constants (#34063): @potiuk @uranusjr Linked issues:
  • Add state utils to Public Airflow Interface (#34059)
• [x] Fix for TaskGroup toggles for duplicated labels (#34072): @Usiel Linked issues:
  • Toggling TaskGroup toggles all TaskGroups with the same label on Graph/Grid (#34066)
• [x] Add missing audit logs for default flask action: add, edit and delete (#34090): @Raul824 @hussein-awala Linked issues:
  • Delete record of dag run is not being logged in audit logs when clicking on delete button in list dag runs (#34069)
• [ ] Refactor: Consolidate import and usage of random (#34108): @eumiro
• [ ] Fix issues related to access_control={} (#34114): @SamWheating
• [ ] Refactor: Simplify string generation (#34118): @eumiro
• [x] Fix unfound ab_user table in the CLI session (#34120): @mbarugelCA @hussein-awala Linked issues:
  • sqlalchemy error when running CLI command airflow tasks test (#34109)
• [x] Fix the required permissions to clear a TI from the UI (#34123): @Raul824 @hussein-awala Linked issues:
  • Can Delete on Task Instances permission is required to clear task (#34092)
• [x] Ensure details panel is shown when any tab is selected (#34136): @Usiel
• [x] Fix FAB-related logging format interpolation (#34139): @Taragolis
• [ ] Fix query bug in next_run_datasets_summary endpoint (#34143): @ephraimbuddy
• [ ] correcting wrong time showing in grid view (#34179): @Abhishek-kumar-samsung
• [ ] Refactor integer division (#34180): @eumiro
• [ ] Refactor: Simplify comparisons (#34181): @eumiro
• [ ] Only tag :latest image when default python version is used (#34182): @potiuk
• [x] Move default timezone to except block (#34245): @hussein-awala
• [ ] Make param validation consistent for DAG validation and triggering (#34248): @jens-scheffler-bosch @ldacey Linked issues:
  • Params Definition is Required to be JSON Schema Valid also if Manually Triggered (#31299)
  • 100+ DAGs fail to import after update to 2.7.1 due to "params without default values" (#34227)
  • Fix handling of default value and serialization of Param class (#33141)
• [x] Replace assert by if...raise in www package (#34249): @hussein-awala
• [x] Replace assert by if...raise in decorators package (#34250): @hussein-awala
• [ ] Change "not all" to "any" for ease of readability (#34259): @pankajkoti
• [ ] Set loglevel=DEBUG in 'Not syncing DAG-level permissions' (#34268): @csp33
• [ ] Fix www cluster_activity view not loading due to standaloneDagProcessor templating (#34274): @erdos2n @yermalov-here Linked issues:
  • Hide Irrelevant Dag Processor from Cluster Activity Page (#33611)
• [x] Update min-sqlalchemy version to account for latest features used (#34293): @potiuk @Taragolis Linked issues:
  • Use NOT EXISTS subquery instead of tuple_not_in_condition (#33527)
• [ ] REST API: Fix task instance access issue in the batch endpoint (#34315): @ephraimbuddy
• [ ] Fixed rows count in the migration script (#34348): @alartum
• [ ] Refactor os.path.splitext to Path.* (#34352): @eumiro
• [x] Fix dag warning endpoint permissions (#34355): @hussein-awala
• [x] Deprecate numeric type python version in PythonVirtualEnvOperator (#34359): @Xiroo
• [x] Fix datetime deserialization by replacing timezone short code by its name in serialized values (#34492): @hussein-awala
• [ ] Change Hide/Show Details button title in Grid View (#34519): @yiannis-had
• [ ] Add more exemptions to lengthy metric list (#34531): @sa1 Linked issues:
  • Airflow DAG fails to run if dag_id + task_id is too long with OTEL integration enabled. (#34416)
• [x] Fix: make dry run optional for patch task instance (#34568): @Calder-Ty Linked issues:
  • DryRun is not optional for patch task instance (#34563)
• [x] Import Dataset from airflow.datasets to avoid circular dependency (#34586): @drobert-bfm @hussein-awala Linked issues:
  • Circular dependency issue when trying to locally test DAG (#34558)
• [x] Fix is_parent_mapped value by checking if any of the parent tg is mapped (#34587): @fritz-astronomer @hussein-awala Linked issues:
  • Unable to retrieve logs for nested task group when parent is mapped (#34535)
• [ ] Fail dag test if defer without triggerer (#34619): @dstandish
• [x] Use iterative loop to look for mapped parent (#34622): @uranusjr @hussein-awala Linked issues:
  • Fix is_parent_mapped value by checking if any of the parent tg is mapped (#34587)
• [ ] Don't run isort when creating new alembic migrations (#34636): @jedcunningham
• [ ] Standardise output of "airflow connections export" to those of "airflow variables export" and "airflow users export" (#34640): @gbonazzoli @Jayden-Chiu Linked issues:
  • Standardise output of "airflow connections export" to those of "airflow variables export" and "airflow users export" (#34232)
• [x] Restrict astroid version < 3 (#34658): @Taragolis
• [ ] Add support for ZoneInfo and generic UTC (#34683): @bolkedebruin
• [ ] misc: Fix spelling for SesssionExemptMixin (#34696): @dkalamar
• [x] Check if the lower of provided values are sensitives in config endpoint (#34712): @hussein-awala
• [ ] Order cherrypicking candidates by the closed date (#34739): @ephraimbuddy
• [x] Fix autodetect_docker_context for list of dict case (#34779): @utkarsharma2
• [ ] zoneinfo.ZoneInfo is not available on < Python 3.9 (#34804): @bolkedebruin
• [ ] Add explanation about audit log users to security model (#34814): @potiuk

Thanks to all who contributed to the release (probably not a complete list!): @dkalamar @jedcunningham @Raul824 @yiannis-had @Xiroo @erdos2n @drobert-bfm @dstandish @ephraimbuddy @Usiel @potiuk @hussein-awala @eumiro @mbarugelCA @jens-scheffler-bosch @utkarsharma2 @yermalov-here @Taragolis @Abhishek-kumar-samsung @csp33 @pankajkoti @mj-dd @gbonazzoli @fritz-astronomer @sa1 @ldacey @SamWheating @risicle @Calder-Ty @Bisk1 @bolkedebruin @uranusjr @alartum @Jayden-Chiu

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jedcunningham]

34011 and #34636 both should have been skipped - providers and dev env respectively.

[ephraimbuddy]

34011 and #34636 both should have been skipped - providers and dev env respectively.

Mistake

[Calder-Ty]

34568 Is working as expected

[utkarsharma2]

Tested #34779 https://github.com/apache/airflow/pull/34779, working as expected.

Thanks, Utkarsh Sharma

On Tue, Oct 10, 2023 at 7:54 PM Tyler Calder @.***> wrote:

34568 https://github.com/apache/airflow/pull/34568 Is working as

expected

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/34830#issuecomment-1755534215, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDLAHOCBOWAFK7D2GOHDKTX6VLDLAVCNFSM6AAAAAA5YNKLICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVGUZTIMRRGU . You are receiving this because you were mentioned.Message ID: @.***>

[Xiroo]

#34359 is working as expected.

[Bisk1]

33746, #33893 and #34027 work as expected

[Usiel]

https://github.com/apache/airflow/pull/34072 and https://github.com/apache/airflow/pull/34136 work as expected.

I also tested a few other PRs:

• Works as expected: https://github.com/apache/airflow/pull/34090
• Does not work for me: https://github.com/apache/airflow/pull/34120 (details below)

Details on #34120 issue (cc @hussein-awala)

Tested both - locally with Breeze (Posgres 11, Python 3.8, git rev `c8b25cb3ee` aka `2.7.2rc1` tag) and - on a test environment (MySQL 5.7, Python 3.9, PyPi package 2.7.2rc1). In both cases, for different DAGs, I get the error below. https://github.com/apache/airflow/pull/34656 seems to have broken this for me. ``` [2023-10-11T02:22:36.229+0000] {cli_action_loggers.py:85} DEBUG - Calling callbacks: [] Traceback (most recent call last): File "/opt/.pyenv/versions/3.9.8/envs/airflow/bin/airflow", line 8, in sys.exit(main()) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/__main__.py", line 60, in main args.func(args) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/cli/cli_config.py", line 49, in command return func(*args, **kwargs) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/utils/cli.py", line 113, in wrapper return f(*args, **kwargs) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 634, in task_test session.delete(ti.dag_run) File "/opt/.pyenv/versions/3.9.8/lib/python3.9/contextlib.py", line 126, in __exit__ next(self.gen) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/utils/session.py", line 37, in create_session session.commit() File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1454, in commit self._transaction.commit(_to_root=self.future) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 832, in commit self._prepare_impl() File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 811, in _prepare_impl self.session.flush() File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3449, in flush self._flush(objects) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3589, in _flush transaction.rollback(_capture_exception=True) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ compat.raise_( File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush flush_context.execute() File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute rec.execute(self) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 667, in execute util.preloaded.orm_persistence.delete_obj( File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 330, in delete_obj table_to_mapper = base_mapper._sorted_tables File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1184, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3386, in _sorted_tables sorted_ = sql_util.sort_tables( File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1217, in sort_tables for (t, fkcs) in sort_tables_and_constraints( File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1289, in sort_tables_and_constraints filtered = filter_fn(fkc) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1207, in _skip_fn if skip_fn(fk): File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3369, in skip dep = table_to_mapper.get(fk.column.table) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2532, in column return self._resolve_column() File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2543, in _resolve_column raise exc.NoReferencedTableError( sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'dag_run_note.user_id' could not find table 'ab_user' with which to generate a foreign key to target column 'id' [2023-10-11T02:22:36.234+0000] {settings.py:353} DEBUG - Disposing DB connection pool (PID 5345) ```

[ephraimbuddy]

34072 and #34136 work as expected.

I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

[hussein-awala]

34072 and #34136 work as expected.

I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

Yes it should not considered as blocker, but if there is a different blocker for the RC (I hope not), we can cherry-pick #34656 in the next RC

[hussein-awala]

I tested my changes, they are present in the RC. I tested only the bug fixes, and all looks good, except #34120 as discussed above.

[ephraimbuddy]

34072 and #34136 work as expected.

I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

Yes it should not considered as blocker, but if there is a different blocker for the RC (I hope not), we can cherry-pick #34656 in the next RC

34656 is in the release, but not on the release note. You can see the milestone and changelog:skip label on it. I wanted to combine it with #34120 on the release note but forgot that.

We probably need another fix. Also, you can go to the v2-7-stable to confirm if the change is there

[jscheffl]

Checked for #34248 and can confirm that (1) code is correctly merged and (2) fix is working! Thanks for making the release!

[ephraimbuddy]

Airflow 2.7.2 has been released. Thank you all for testing the release candidate

[yermalov-here]

34274 works as expected

[sa1]

Only verified after release but #34531 is working.