pierrejeambrun commented 1 year ago

We have a kind request for all the contributors to the latest Apache Airflow RC 2.5.2rc2.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

[ ] Fix inconsitencies in checking edit permissions for a DAG (#20346): @kaxil
[ ] logging poke info when external dag is not none and task_id and task_ids are none (#28097): @surabathini
[x] Migrate remaining core sensors tests to pytest (#28204): @potiuk @Taragolis Linked issues:
- Migrate remaining core sensors tests to pytest (#28177)
- Revert "Migrate remaining core sensors tests to pytest (#28177)" (#28196)
[ ] Fix manual task trigger failing for k8s. (#28394): @snjypl
[x] Fix bugs introduced when moving the code of issue generation to breeze (#28683): @potiuk Linked issues:
- Move provider issue generation to breeze (#28352)
[ ] introduce dag processor job fix #27140 (#28799): @farhan0syakir
[ ] Throttle streaming log reads (#28818): @dstandish
[ ] Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824): @dependabot[bot]
[ ] Annotate KubeExecutor pods that we don't delete (#28844): @jedcunningham
[ ] KubenetesExecutor sends state even when successful (#28871): @jedcunningham
[ ] Add dep context description for better log message (#28875): @dstandish
[ ] Be more selective when adopting pods with KubernetesExecutor (#28899): @jedcunningham
[ ] Allow URI without authority and host blocks in airflow connections add (#28922): @Taragolis
[ ] Update how PythonSensor returns values from python_callable (#28932): @SoxMax
[ ] Resolve all variables in pickled XCom iterator (#28982): @uranusjr
[ ] Write action log to DB when DAG run is trigged via API (#28998): @kolfild26
[ ] Adding dataset to be discovered by static analyzers (#29007): @bharat99k
[ ] Fix leak sensitive field via V1EnvVar on exception (#29016): @ecerulm
[x] Sanitize url_for arguments before they are passed (#29039): @potiuk
[x] Clarify graph view dynamic task labels (#29042): @bbovenzi
[x] Fix kerberos authentication for the REST API. (#29054): @potiuk
[x] Handling error on cluster policy itself (#29056): @okayhooni
[x] Check for run_id url param when linking to graph/gantt views (#29066): @bbovenzi
[x] Emit dagrun duration when it fails because of a timeout (#29076): @yangguoaws @hussein-awala Linked issues:
- Metrics dagrun.duration.failed. not updated when the dag run failed due to timeout (#29013)
[ ] Remove upper bound limitation for pytest (#29086): @auvipy
[ ] Fix warning in migrations about old config. (#29092): @ashb
[x] Make options output of breeze command consistent and complete (#29097): @potiuk
[ ] Fix to run tasks on Dask Worker for Windows systems (#29107): @casra-developers
[ ] Fix access to context in functions decorated by task.sensor (#29146): @Thomas-McKanna Linked issues:
- Fix access to context in functions decorated by task.sensor (#29137)
[ ] Bump ua-parser-js from 0.7.31 to 0.7.33 in /airflow/www (#29172): @dependabot[bot]
[ ] Fixing Task Duration view in case of manual DAG runs only (#22015) (#29195): @duke8585
[x] Fix Trigger DAG w/config origin url (#29212): @hussein-awala @enriqueayala Linked issues:
- Trigger DAG w/config raising error from task detail views (#29197)
[ ] Fix dag run trigger with a note. (#29228): @vedantlodha @ttben Linked issues:
- Bad request when triggering dag run with note in payload (#28825)
[x] Clarify description of worker control config (#29247): @potiuk
[x] Increase max height for grid view elements (#29367): @bbovenzi
[ ] Fix grid logs for large logs (#29390): @bbovenzi
[x] Fix validation of date-time field in API and Parameter schemas (#29395): @potiuk
[x] Persist DAG and task doc values in TaskFlow API if explicitly set (#29399): @josh-fell
[ ] Ensure Serialized DAG is deleted (#29407): @michaelmicheal
[ ] Fix Rest API update user output (#29409): @vincbeck
[x] Edgemodifier refactoring w/ labels in TaskGroup edge case (#29410): @vchiapaikeo
[ ] datasets, next_run_datasets, remove unnecessary timestamp filter (#29441): @tseruga @michaelmicheal Linked issues:
- Dataset Next Trigger Modal Not Populating Latest Update (#26892)
[x] Fix nested fields rendering in mapped operators (#29451): @pshrivastava27 @hussein-awala @bdsoha Linked issues:
- Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator (#29432)
- Mapped KubernetesPodOperater not rendering nested templates (#25588)
[x] Make prev_logical_date variable offset-aware (#29454): @akrava
[ ] Use newer setuptools v67.2.0 (#29465): @lewijw @arjunanan6 Linked issues:
- Require newer version of pypi/setuptools to remove security scan issue (CVE-2022-40897) (#29428)
[x] Require explicit flag to remove MyPy cache when running breeze stop (#29493): @potiuk Linked issues:
- Also remove mypy cache when stopping breeze (#29184)
[x] Change permissions of config/password files created by airflow (#29495): @potiuk
[x] Do not show version/node in UI traceback for unauthenticated user (#29501): @potiuk
[x] Remove section check from get-value command (#29541): @Limess @potiuk Linked issues:
- Docker image fails to start if celery config section is not defined (#29537)
[x] Don't display empty rendered attrs in Task Instance Details page (#29545): @josh-fell @lorenzomagni Linked issues:
- Hide non-used docs attributes from Task Instance Detail (#29515)
[ ] Change expose_hostname default to false (#29547): @jedcunningham
[ ] Bump undici from 5.9.1 to 5.19.1 in /airflow/www (#29583): @dependabot[bot]
[x] fix clear dag run openapi spec responses by adding additional return type (#29600): @aliotta
[x] convert moment with timezone to UTC instead of raising an exception (#29606): @Gollum999 @hussein-awala Linked issues:
- DateTimeSensorAsync breaks if target_time is timezone-aware (#29576)
[x] Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645): @ephraimbuddy
[x] Move extra links position in grid view (#29703): @bbovenzi
[x] Fix on_failure_callback when task receives a SIGTERM (#29743): @ephraimbuddy
[x] DAG list sorting lost when switching page (#29756): @alejandrofm @amoghrajesh Linked issues:
- DAG list, sorting lost when switching page (#29322)
[x] Improve DAG loading in Graph view when using Dynamic Tasks (#29791): @Josersanvil @jose-workpath Linked issues:
- DAG loading very slow in Graph view when using Dynamic Tasks (#27483)
[x] Add a check for not templateable fields (#29821): @hussein-awala @cosinequanon Linked issues:
- DAG fails serialization if template_field contains execution_timeout (#29819)
[x] Updating dag run endpoint to 404 if requested dag is inactive (#29860): @SamWheating
[ ] Update min version of python-deamon to fix containerd file limits (#29916): @antonio-antuan @potiuk Linked issues:
- high memory leak, cannot start even webserver (#29841)
[ ] Limit importlib-metadata backport to < 5.0.0 (#29924, #30069) @potiuk

Thanks to all who contributed to the release (probably not a complete list!): @Gollum999 @snjypl @auvipy @amoghrajesh @antonio-antuan @bharat99k @vchiapaikeo @SamWheating @jose-workpath @tseruga @akrava @Josersanvil @Thomas-McKanna @ashb @SoxMax @bdsoha @vedantlodha @lewijw @hussein-awala @arjunanan6 @dependabot[bot] @yangguoaws @kaxil @Limess @surabathini @jedcunningham @dstandish @pshrivastava27 @ecerulm @ttben @casra-developers @josh-fell @duke8585 @ephraimbuddy @michaelmicheal @farhan0syakir @okayhooni @lorenzomagni @bbovenzi @kolfild26 @potiuk @aliotta @alejandrofm @cosinequanon @enriqueayala @uranusjr @vincbeck @Taragolis

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

auvipy commented 1 year ago

how should I report?

amoghrajesh commented 1 year ago

@pierrejeambrun what is the best way to test this?

pierrejeambrun commented 1 year ago

@auvipy you can directly report in this issue your findings. And if the change (PR) you tested is working as expected or not. (Or any other issue you noticed in the release candidate)

@amoghrajesh The easiest way is to install the release candidate in a fresh virtualenv, then run airflow in standalone mode and test changes there. (If you have a dev environment for airflow you can also upgrade there and see if everything is working as expected)

vchiapaikeo commented 1 year ago

https://github.com/apache/airflow/pull/29410 is still looking good.

We did see an unrelated issue from https://github.com/apache/airflow/pull/29821 but that seems to be a user error on our end that we need to correct. Thank you @hussein-awala for that fix!

amoghrajesh commented 1 year ago

https://github.com/apache/airflow/pull/29756 looks good on the RC build. Performed small multiple tests, one snippet attached to this comment. @pierrejeambrun

https://user-images.githubusercontent.com/35884252/224530091-8ecfadd3-c5a1-4607-a208-13068e7f07f2.mov

potiuk commented 1 year ago

28204, #28683, #29097. #29493, - were just CI/test modifications
29039 looks good (sanitization is there)
29247 - updated description is there
29054 - I'd appreciate @BMFH if you could check if it works with your kerberos installation

To be continued (have to hurry now)

potiuk commented 1 year ago

29395 - correctly validates badly formatted time
29495 - is good:

Both standard config and standalone password file have user-only permissions:

^Xroot@f8d4a86cf3ef:~/airflow# ls -la airflow.cfg
-rw------- 1 root root 51764 Mar 12 11:36 airflow.cfg

⌁ ⌂23.88 [jarek:~/airflow] [airflow-astroid] % ls -la standalone_admin_password.txt
-rw-------  1 jarek  staff  16 Mar 12 12:48 standalone_admin_password.txt

29501 - the details are redacted when user is not authenticated

29541 works fine when no celery section is defined in cfg we get now:
```
> airflow config get-value celery broker_url
redis://redis:6379/0
```

rather than exit 1 and:

> airflow config get-value celery broker_url
The section [celery] is not found in config.

29916 - I've checked that python-daemon installed is 3.0.1 which is good (also to note 3.0.0 that fixed the problem had been yanked by the maintainer as it did not work without setuptools (and it uses packaging now instead). But that should keep the fix released in 3.0.0. @antonio-antuan maybe you could check if the 2.5.2.rc1 fixes the memory issue in your setup (I have no access to my test machine I set it up for testin and travelling for the next few days).

potiuk commented 1 year ago

All looks good for my fixes - except the two issues where it "looks" good but I cannot test it easily and I would love to get confirmation :)

hussein-awala commented 1 year ago

I've tested #29076, #29212, #29451, #29606, #29821 and they all look good.

BMFH commented 1 year ago

I've checked #29054. It works fine.

Josersanvil commented 1 year ago

Tested #29791 in the 2.5.2rc1 docker tag and it looks good 🙂

josh-fell commented 1 year ago

29399 and #29545 look good!

bbovenzi commented 1 year ago

29066, #29703, #29367, #29042, and #29600 look good

SamWheating commented 1 year ago

Tested https://github.com/apache/airflow/pull/29860 in breeze after pulling the 2.5.2rc1 tag, looks like it works as expected.

Tried to create a DAGRun from an inactive DAG, and got a 404 rather than a 500:


curl -X POST localhost:8080/api/v1/dags/data_warehouse_dag_o259owoo3s/dagRuns -d '{"dag_run_id":"string2","logical_date":"2019-08-24T14:15:24Z","execution_date":"2019-08-24T14:15:24Z","conf":{},"note":"strings"}'  -H 'Content-Type: application/json'

{
  "detail": "DAG with dag_id: 'data_warehouse_dag_o259owoo3s' not found",
  "status": 404,
  "title": "DAG not found", 
  "type": "https://airflow.apache.org/docs/apache-airflow/2.5.2/stable-rest-api-ref.html#section/Errors/NotFound"                       
}

pierrejeambrun commented 1 year ago

A bug was found in rc1 that necessitate an rc2 (constraints). I think we can safely assume that what was tested for rc1 is still working in rc2. Feel free to do more testing. The issue has been updated to target 2.5.2rc2