apache/airflow (apache/airflow)
### [`v2.7.1`](https://togithub.com/apache/airflow/blob/HEAD/RELEASE_NOTES.rst#Airflow-271-2023-09-07)
[Compare Source](https://togithub.com/apache/airflow/compare/2.7.0...2.7.1)
Significant Changes
^^^^^^^^^^^^^^^^^^^
CronTriggerTimetable is now less aggressive when trying to skip a run ([#33404](https://togithub.com/apache/airflow/issues/33404))
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
When setting `catchup=False`, CronTriggerTimetable no longer skips a run if
the scheduler does not query the timetable immediately after the previous run
has been triggered.
This should not affect scheduling in most cases, but can change the behaviour if
a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with
`catchup=False`) would only start a run after a DAG is unpaused, but with this
change, the scheduler would try to look at little bit back to schedule the
previous run that covers a part of the period when the DAG was paused. This
means you will need to keep a DAG paused longer (namely, for the entire cron
period to pass) to really skip a run.
Note that this is also the behaviour exhibited by various other cron-based
scheduling tools, such as `anacron`.
`conf.set()` becomes case insensitive to match `conf.get()` behavior ([#33452](https://togithub.com/apache/airflow/issues/33452))
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Also, `conf.get()` will now break if used with non-string parameters.
`conf.set(section, key, value)` used to be case sensitive, i.e. `conf.set("SECTION", "KEY", value)`
and `conf.set("section", "key", value)` were stored as two distinct configurations.
This was inconsistent with the behavior of `conf.get(section, key)`, which was always converting the section and key to lower case.
As a result, configuration options set with upper case characters in the section or key were unreachable.
That's why we are now converting section and key to lower case in `conf.set` too.
We also changed a bit the behavior of `conf.get()`. It used to allow objects that are not strings in the section or key.
Doing this will now result in an exception. For instance, `conf.get("section", 123)` needs to be replaced with `conf.get("section", "123")`.
Bug Fixes
"""""""""
- Ensure that tasks wait for running indirect setup ([#33903](https://togithub.com/apache/airflow/issues/33903))
- Respect "soft_fail" for core async sensors ([#33403](https://togithub.com/apache/airflow/issues/33403))
- Differentiate 0 and unset as a default param values ([#33965](https://togithub.com/apache/airflow/issues/33965))
- Raise 404 from Variable PATCH API if variable is not found ([#33885](https://togithub.com/apache/airflow/issues/33885))
- Fix `MappedTaskGroup` tasks not respecting upstream dependency ([#33732](https://togithub.com/apache/airflow/issues/33732))
- Add limit 1 if required first value from query result ([#33672](https://togithub.com/apache/airflow/issues/33672))
- Fix UI DAG counts including deleted DAGs ([#33778](https://togithub.com/apache/airflow/issues/33778))
- Fix cleaning zombie RESTARTING tasks ([#33706](https://togithub.com/apache/airflow/issues/33706))
- `SECURITY_MANAGER_CLASS` should be a reference to class, not a string ([#33690](https://togithub.com/apache/airflow/issues/33690))
- Add back `get_url_for_login` in security manager ([#33660](https://togithub.com/apache/airflow/issues/33660))
- Fix `2.7.0 db` migration job errors ([#33652](https://togithub.com/apache/airflow/issues/33652))
- Set context inside templates ([#33645](https://togithub.com/apache/airflow/issues/33645))
- Treat dag-defined access_control as authoritative if defined ([#33632](https://togithub.com/apache/airflow/issues/33632))
- Bind engine before attempting to drop archive tables ([#33622](https://togithub.com/apache/airflow/issues/33622))
- Add a fallback in case no first name and last name are set ([#33617](https://togithub.com/apache/airflow/issues/33617))
- Sort data before `groupby` in TIS duration calculation ([#33535](https://togithub.com/apache/airflow/issues/33535))
- Stop adding values to rendered templates UI when there is no dagrun ([#33516](https://togithub.com/apache/airflow/issues/33516))
- Set strict to True when parsing dates in webserver views ([#33512](https://togithub.com/apache/airflow/issues/33512))
- Use `dialect.name` in custom SA types ([#33503](https://togithub.com/apache/airflow/issues/33503))
- Do not return ongoing dagrun when a `end_date` is less than `utcnow` ([#33488](https://togithub.com/apache/airflow/issues/33488))
- Fix a bug in `formatDuration` method ([#33486](https://togithub.com/apache/airflow/issues/33486))
- Make `conf.set` case insensitive ([#33452](https://togithub.com/apache/airflow/issues/33452))
- Allow timetable to slightly miss catchup cutoff ([#33404](https://togithub.com/apache/airflow/issues/33404))
- Respect `soft_fail` argument when `poke` is called ([#33401](https://togithub.com/apache/airflow/issues/33401))
- Create a new method used to resume the task in order to implement specific logic for operators ([#33424](https://togithub.com/apache/airflow/issues/33424))
- Fix DagFileProcessor interfering with dags outside its `processor_subdir` ([#33357](https://togithub.com/apache/airflow/issues/33357))
- Remove the unnecessary ` ` text in Provider's view ([#33326](https://togithub.com/apache/airflow/issues/33326))
- Respect `soft_fail` argument when ExternalTaskSensor runs in deferrable mode ([#33196](https://togithub.com/apache/airflow/issues/33196))
- Fix handling of default value and serialization of Param class ([#33141](https://togithub.com/apache/airflow/issues/33141))
- Check if the dynamically-added index is in the table schema before adding ([#32731](https://togithub.com/apache/airflow/issues/32731))
- Fix rendering the mapped parameters when using `expand_kwargs` method ([#32272](https://togithub.com/apache/airflow/issues/32272))
- Fix dependencies for celery and opentelemetry for Python 3.8 ([#33579](https://togithub.com/apache/airflow/issues/33579))
Misc/Internal
"""""""""""""
- Bring back `Pydantic` 1 compatibility ([#34081](https://togithub.com/apache/airflow/issues/34081), [#33998](https://togithub.com/apache/airflow/issues/33998))
- Use a trimmed version of README.md for PyPI ([#33637](https://togithub.com/apache/airflow/issues/33637))
- Upgrade to `Pydantic` 2 ([#33956](https://togithub.com/apache/airflow/issues/33956))
- Reorganize `devel_only` extra in Airflow's setup.py ([#33907](https://togithub.com/apache/airflow/issues/33907))
- Bumping `FAB` to `4.3.4` in order to fix issues with filters ([#33931](https://togithub.com/apache/airflow/issues/33931))
- Add minimum requirement for `sqlalchemy to 1.4.24` ([#33892](https://togithub.com/apache/airflow/issues/33892))
- Update version_added field for configs in config file ([#33509](https://togithub.com/apache/airflow/issues/33509))
- Replace `OrderedDict` with plain dict ([#33508](https://togithub.com/apache/airflow/issues/33508))
- Consolidate import and usage of itertools ([#33479](https://togithub.com/apache/airflow/issues/33479))
- Static check fixes ([#33462](https://togithub.com/apache/airflow/issues/33462))
- Import utc from datetime and normalize its import ([#33450](https://togithub.com/apache/airflow/issues/33450))
- D401 Support ([#33352](https://togithub.com/apache/airflow/issues/33352), [#33339](https://togithub.com/apache/airflow/issues/33339), [#33337](https://togithub.com/apache/airflow/issues/33337), [#33336](https://togithub.com/apache/airflow/issues/33336), [#33335](https://togithub.com/apache/airflow/issues/33335), [#33333](https://togithub.com/apache/airflow/issues/33333), [#33338](https://togithub.com/apache/airflow/issues/33338))
- Fix some missing type hints ([#33334](https://togithub.com/apache/airflow/issues/33334))
- D205 Support - Stragglers ([#33301](https://togithub.com/apache/airflow/issues/33301), [#33298](https://togithub.com/apache/airflow/issues/33298), [#33297](https://togithub.com/apache/airflow/issues/33297))
- Refactor: Simplify code ([#33160](https://togithub.com/apache/airflow/issues/33160), [#33270](https://togithub.com/apache/airflow/issues/33270), [#33268](https://togithub.com/apache/airflow/issues/33268), [#33267](https://togithub.com/apache/airflow/issues/33267), [#33266](https://togithub.com/apache/airflow/issues/33266), [#33264](https://togithub.com/apache/airflow/issues/33264), [#33292](https://togithub.com/apache/airflow/issues/33292), [#33453](https://togithub.com/apache/airflow/issues/33453), [#33476](https://togithub.com/apache/airflow/issues/33476), [#33567](https://togithub.com/apache/airflow/issues/33567),
[#33568](https://togithub.com/apache/airflow/issues/33568), [#33480](https://togithub.com/apache/airflow/issues/33480), [#33753](https://togithub.com/apache/airflow/issues/33753), [#33520](https://togithub.com/apache/airflow/issues/33520), [#33623](https://togithub.com/apache/airflow/issues/33623))
- Fix `Pydantic` warning about `orm_mode` rename ([#33220](https://togithub.com/apache/airflow/issues/33220))
- Add MySQL 8.1 to supported versions. ([#33576](https://togithub.com/apache/airflow/issues/33576))
- Remove `Pydantic` limitation for version < 2 ([#33507](https://togithub.com/apache/airflow/issues/33507))
Doc only changes
"""""""""""""""""
- Add documentation explaining template_ext (and how to override it) ([#33735](https://togithub.com/apache/airflow/issues/33735))
- Explain how users can check if python code is top-level ([#34006](https://togithub.com/apache/airflow/issues/34006))
- Clarify that DAG authors can also run code in DAG File Processor ([#33920](https://togithub.com/apache/airflow/issues/33920))
- Fix broken link in Modules Management page ([#33499](https://togithub.com/apache/airflow/issues/33499))
- Fix secrets backend docs ([#33471](https://togithub.com/apache/airflow/issues/33471))
- Fix config description for base_log_folder ([#33388](https://togithub.com/apache/airflow/issues/33388))
Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
[ ] If you want to rebase/retry this PR, check this box
This PR has been generated by Mend Renovate. View repository job log here.
This PR contains the following updates:
2.7.0->2.7.1Release Notes
apache/airflow (apache/airflow)
### [`v2.7.1`](https://togithub.com/apache/airflow/blob/HEAD/RELEASE_NOTES.rst#Airflow-271-2023-09-07) [Compare Source](https://togithub.com/apache/airflow/compare/2.7.0...2.7.1) Significant Changes ^^^^^^^^^^^^^^^^^^^ CronTriggerTimetable is now less aggressive when trying to skip a run ([#33404](https://togithub.com/apache/airflow/issues/33404)) """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" When setting `catchup=False`, CronTriggerTimetable no longer skips a run if the scheduler does not query the timetable immediately after the previous run has been triggered. This should not affect scheduling in most cases, but can change the behaviour if a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with `catchup=False`) would only start a run after a DAG is unpaused, but with this change, the scheduler would try to look at little bit back to schedule the previous run that covers a part of the period when the DAG was paused. This means you will need to keep a DAG paused longer (namely, for the entire cron period to pass) to really skip a run. Note that this is also the behaviour exhibited by various other cron-based scheduling tools, such as `anacron`. `conf.set()` becomes case insensitive to match `conf.get()` behavior ([#33452](https://togithub.com/apache/airflow/issues/33452)) """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Also, `conf.get()` will now break if used with non-string parameters. `conf.set(section, key, value)` used to be case sensitive, i.e. `conf.set("SECTION", "KEY", value)` and `conf.set("section", "key", value)` were stored as two distinct configurations. This was inconsistent with the behavior of `conf.get(section, key)`, which was always converting the section and key to lower case. As a result, configuration options set with upper case characters in the section or key were unreachable. That's why we are now converting section and key to lower case in `conf.set` too. We also changed a bit the behavior of `conf.get()`. It used to allow objects that are not strings in the section or key. Doing this will now result in an exception. For instance, `conf.get("section", 123)` needs to be replaced with `conf.get("section", "123")`. Bug Fixes """"""""" - Ensure that tasks wait for running indirect setup ([#33903](https://togithub.com/apache/airflow/issues/33903)) - Respect "soft_fail" for core async sensors ([#33403](https://togithub.com/apache/airflow/issues/33403)) - Differentiate 0 and unset as a default param values ([#33965](https://togithub.com/apache/airflow/issues/33965)) - Raise 404 from Variable PATCH API if variable is not found ([#33885](https://togithub.com/apache/airflow/issues/33885)) - Fix `MappedTaskGroup` tasks not respecting upstream dependency ([#33732](https://togithub.com/apache/airflow/issues/33732)) - Add limit 1 if required first value from query result ([#33672](https://togithub.com/apache/airflow/issues/33672)) - Fix UI DAG counts including deleted DAGs ([#33778](https://togithub.com/apache/airflow/issues/33778)) - Fix cleaning zombie RESTARTING tasks ([#33706](https://togithub.com/apache/airflow/issues/33706)) - `SECURITY_MANAGER_CLASS` should be a reference to class, not a string ([#33690](https://togithub.com/apache/airflow/issues/33690)) - Add back `get_url_for_login` in security manager ([#33660](https://togithub.com/apache/airflow/issues/33660)) - Fix `2.7.0 db` migration job errors ([#33652](https://togithub.com/apache/airflow/issues/33652)) - Set context inside templates ([#33645](https://togithub.com/apache/airflow/issues/33645)) - Treat dag-defined access_control as authoritative if defined ([#33632](https://togithub.com/apache/airflow/issues/33632)) - Bind engine before attempting to drop archive tables ([#33622](https://togithub.com/apache/airflow/issues/33622)) - Add a fallback in case no first name and last name are set ([#33617](https://togithub.com/apache/airflow/issues/33617)) - Sort data before `groupby` in TIS duration calculation ([#33535](https://togithub.com/apache/airflow/issues/33535)) - Stop adding values to rendered templates UI when there is no dagrun ([#33516](https://togithub.com/apache/airflow/issues/33516)) - Set strict to True when parsing dates in webserver views ([#33512](https://togithub.com/apache/airflow/issues/33512)) - Use `dialect.name` in custom SA types ([#33503](https://togithub.com/apache/airflow/issues/33503)) - Do not return ongoing dagrun when a `end_date` is less than `utcnow` ([#33488](https://togithub.com/apache/airflow/issues/33488)) - Fix a bug in `formatDuration` method ([#33486](https://togithub.com/apache/airflow/issues/33486)) - Make `conf.set` case insensitive ([#33452](https://togithub.com/apache/airflow/issues/33452)) - Allow timetable to slightly miss catchup cutoff ([#33404](https://togithub.com/apache/airflow/issues/33404)) - Respect `soft_fail` argument when `poke` is called ([#33401](https://togithub.com/apache/airflow/issues/33401)) - Create a new method used to resume the task in order to implement specific logic for operators ([#33424](https://togithub.com/apache/airflow/issues/33424)) - Fix DagFileProcessor interfering with dags outside its `processor_subdir` ([#33357](https://togithub.com/apache/airflow/issues/33357)) - Remove the unnecessary `` text in Provider's view ([#33326](https://togithub.com/apache/airflow/issues/33326)) - Respect `soft_fail` argument when ExternalTaskSensor runs in deferrable mode ([#33196](https://togithub.com/apache/airflow/issues/33196)) - Fix handling of default value and serialization of Param class ([#33141](https://togithub.com/apache/airflow/issues/33141)) - Check if the dynamically-added index is in the table schema before adding ([#32731](https://togithub.com/apache/airflow/issues/32731)) - Fix rendering the mapped parameters when using `expand_kwargs` method ([#32272](https://togithub.com/apache/airflow/issues/32272)) - Fix dependencies for celery and opentelemetry for Python 3.8 ([#33579](https://togithub.com/apache/airflow/issues/33579)) Misc/Internal """"""""""""" - Bring back `Pydantic` 1 compatibility ([#34081](https://togithub.com/apache/airflow/issues/34081), [#33998](https://togithub.com/apache/airflow/issues/33998)) - Use a trimmed version of README.md for PyPI ([#33637](https://togithub.com/apache/airflow/issues/33637)) - Upgrade to `Pydantic` 2 ([#33956](https://togithub.com/apache/airflow/issues/33956)) - Reorganize `devel_only` extra in Airflow's setup.py ([#33907](https://togithub.com/apache/airflow/issues/33907)) - Bumping `FAB` to `4.3.4` in order to fix issues with filters ([#33931](https://togithub.com/apache/airflow/issues/33931)) - Add minimum requirement for `sqlalchemy to 1.4.24` ([#33892](https://togithub.com/apache/airflow/issues/33892)) - Update version_added field for configs in config file ([#33509](https://togithub.com/apache/airflow/issues/33509)) - Replace `OrderedDict` with plain dict ([#33508](https://togithub.com/apache/airflow/issues/33508)) - Consolidate import and usage of itertools ([#33479](https://togithub.com/apache/airflow/issues/33479)) - Static check fixes ([#33462](https://togithub.com/apache/airflow/issues/33462)) - Import utc from datetime and normalize its import ([#33450](https://togithub.com/apache/airflow/issues/33450)) - D401 Support ([#33352](https://togithub.com/apache/airflow/issues/33352), [#33339](https://togithub.com/apache/airflow/issues/33339), [#33337](https://togithub.com/apache/airflow/issues/33337), [#33336](https://togithub.com/apache/airflow/issues/33336), [#33335](https://togithub.com/apache/airflow/issues/33335), [#33333](https://togithub.com/apache/airflow/issues/33333), [#33338](https://togithub.com/apache/airflow/issues/33338)) - Fix some missing type hints ([#33334](https://togithub.com/apache/airflow/issues/33334)) - D205 Support - Stragglers ([#33301](https://togithub.com/apache/airflow/issues/33301), [#33298](https://togithub.com/apache/airflow/issues/33298), [#33297](https://togithub.com/apache/airflow/issues/33297)) - Refactor: Simplify code ([#33160](https://togithub.com/apache/airflow/issues/33160), [#33270](https://togithub.com/apache/airflow/issues/33270), [#33268](https://togithub.com/apache/airflow/issues/33268), [#33267](https://togithub.com/apache/airflow/issues/33267), [#33266](https://togithub.com/apache/airflow/issues/33266), [#33264](https://togithub.com/apache/airflow/issues/33264), [#33292](https://togithub.com/apache/airflow/issues/33292), [#33453](https://togithub.com/apache/airflow/issues/33453), [#33476](https://togithub.com/apache/airflow/issues/33476), [#33567](https://togithub.com/apache/airflow/issues/33567), [#33568](https://togithub.com/apache/airflow/issues/33568), [#33480](https://togithub.com/apache/airflow/issues/33480), [#33753](https://togithub.com/apache/airflow/issues/33753), [#33520](https://togithub.com/apache/airflow/issues/33520), [#33623](https://togithub.com/apache/airflow/issues/33623)) - Fix `Pydantic` warning about `orm_mode` rename ([#33220](https://togithub.com/apache/airflow/issues/33220)) - Add MySQL 8.1 to supported versions. ([#33576](https://togithub.com/apache/airflow/issues/33576)) - Remove `Pydantic` limitation for version < 2 ([#33507](https://togithub.com/apache/airflow/issues/33507)) Doc only changes """"""""""""""""" - Add documentation explaining template_ext (and how to override it) ([#33735](https://togithub.com/apache/airflow/issues/33735)) - Explain how users can check if python code is top-level ([#34006](https://togithub.com/apache/airflow/issues/34006)) - Clarify that DAG authors can also run code in DAG File Processor ([#33920](https://togithub.com/apache/airflow/issues/33920)) - Fix broken link in Modules Management page ([#33499](https://togithub.com/apache/airflow/issues/33499)) - Fix secrets backend docs ([#33471](https://togithub.com/apache/airflow/issues/33471)) - Fix config description for base_log_folder ([#33388](https://togithub.com/apache/airflow/issues/33388))
Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Mend Renovate. View repository job log here.