ephraimbuddy commented 2 years ago

Body

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.5rc3.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

[ ] Set X-Frame-Options header to DENY unless X_FRAME_ENABLED is set to true (#19491): @subkanthi @planoe-cloudera Linked issues:
- x_frame_enabled logic broken in Airflow 2 (#17255)
[ ] Fix: Do not render undefined graph edges (#19684): @bbovenzi
[ ] Fix Tasks getting stuck in scheduled state (#19747): @vapiravfif @tanelk Linked issues:
- Tasks get stuck in "scheduled" state and starved when dags with huge amount of tasks is scheduled (#19622)
[ ] Rename to_delete to to_cancel in TriggerRunner (#20658): @dstandish
[ ] Fix duplicate trigger creation race condition (#20699): @MatrixManAtYrService @dstandish Linked issues:
- TriggerEvent fires, and then defers a second time (doesn't fire a second time though). (#18392)
- Track completed triggers until deleted from database (#19546)
[ ] Truncate stack trace to DAG user code for exceptions raised during execution (#20731): @malthe
[ ] fix: Update custom connection field processing (#20883): @mike-mcdonald Linked issues:
- Cannot edit custom fields on provider connections (#20839)
[ ] Ensure clear_task_instances sets valid run state (#21116): @uranusjr @uplsh580 Linked issues:
- Running airflow dags backfill --reset-dagruns -s -e results in error when run twice. (#21058)
- Fix 'airflow dags backfill --reset-dagruns' errors when run twice (#21062)
[x] Update ExternalTaskSensorLink to handle templated external_dag_id (#21192): @csp98 @josh-fell Linked issues:
- Webserver "External DAG" button on ExternalTaskSensor not working when dag_id is templated (#21183)
[ ] A trigger might use a connection; make sure we mask passwords (#21207): @malthe
[ ] Log traceback in trigger excs (#21213): @malthe
[ ] Fix trigger dag redirect from task instance log view (#21239): @dstandish
[ ] Fix race condition between triggerer and scheduler (#21316): @malthe
[ ] Reduce DB load incurred by Stale DAG deactivation (#21399): @SamWheating
[ ] Fix max_active_runs=1 not scheduling runs when min_file_process_interval is high (#21413): @ephraimbuddy
[ ] Fix Resources eq check (#21442): @pingzh
[ ] Filter out default configs when overrides exist. (#21539): @xyu @ashb Linked issues:
- PR #18772 breaks sql_alchemy_conn_cmd config (#20092)
- Don't bake ENV and _cmd into tmp config for non-sudo (#18772)
[ ] Fix logging JDBC SQL error when task fails (#21540): @stijn-meersman @hubert-pietron @jyotsa09 Linked issues:
- No SQL error shown when using the JDBC operator (#18482)
- JDBC operator not logging errors (#16295)
[ ] Dispose not used connection pool (#21565): @pingzh
[ ] FIX: extends typing-extensions to be installed with python 3.8+ #21566 (#21567): @frankcash Linked issues:
- typing_extensions package isn't installed with apache-airflow-providers-amazon causing an issue for SqlToS3Operator (#21566)
[ ] Disable default_pool delete on web ui (#21658): @chenglongyan
[ ] Log exception in local executor (#21667): @uranusjr
[ ] Correctly handle multiple '=' in LocalFileSystem secrets. (#21694): @kadai0308 @joeyberkovitz Linked issues:
- LocalFileSystem Secret Backend Bug (#21625)
[ ] Fix stray order_by(TaskInstance.execution_date) (#21705): @uranusjr @TidesMind Linked issues:
- Airflow >= 2.2.0 execution date change is failing TaskInstance get_task_instances method and possibly others (#21656)
[ ] Fix filesystem sensor for directories (#21729): @mixilchenko
[ ] Fix graph autorefresh on page load (#21736): @bbovenzi
[ ] Fix triggerer --capacity parameter (#21753): @thesuperzapper @msumit Linked issues:
- triggerer --capacity parameter does not work (#21752)
[ ] Fix assignment of unassigned triggers (#21770): @jkramer-ginkgo
[ ] Fix incorrect data provided to tries & landing times charts (#21928): @millin
[ ] Fixed dask executor and tests (#22027): @subkanthi @potiuk Linked issues:
- Removes limitations from Dask dependencies (#22017)
[x] Add the new Airflow Trove Classifier to setup.cfg (#22241): @potiuk
[ ] Fix broken links to celery documentation (#22364): @davidhagens
[ ] Add back celery intersphinx mapping (#22370): @kaxil
[x] Replace timedelta.max with year long timdelta in test_manager (#22527): @potiuk
[ ] Check and disallow a relative path for sqlite (#22530): @yenchenLiu

Thanks to all who contributed to the release (probably not a complete list!): @frankcash @yenchenLiu @bbovenzi @SamWheating @ephraimbuddy @csp98 @joeyberkovitz @planoe-cloudera @hubert-pietron @dstandish @stijn-meersman @jyotsa09 @kadai0308 @chenglongyan @pingzh @TidesMind @josh-fell @potiuk @ashb @mike-mcdonald @vapiravfif @thesuperzapper @msumit @xyu @uplsh580 @tanelk @subkanthi @uranusjr @jkramer-ginkgo @millin @malthe @mixilchenko @davidhagens @MatrixManAtYrService @kaxil

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

potiuk commented 2 years ago

All good.

Checked #22241 - > classifier is there

21905 and #22527 were mostly to make sure the tests pass (they did :) )

22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":

    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

mixilchenko commented 2 years ago

https://github.com/apache/airflow/pull/21729 looks good for 2.2.5 release

raphaelauv commented 2 years ago

https://github.com/apache/airflow/pull/18797 look good

with python 3.6

[2022-03-27, 18:45:42 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.2e925545978340a197ae86ee955e660a with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-18T0000000000-90832bb28', 'try_number': '1'}
[2022-03-27, 18:45:42 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:43 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:44 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:45 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:47 UTC] {pod_manager.py:203} INFO - toto
[2022-03-27, 18:46:03 UTC] {pod_manager.py:203} INFO - finish
[2022-03-27, 18:46:04 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:46:04 UTC] {taskinstance.py:1288} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220318T000000, start_date=20220327T184542, end_date=20220327T184604
[2022-03-27, 18:46:04 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:46:04 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

with python 3.9

[2022-03-27, 18:29:26 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-26T0000000000-09f4abc4d', 'try_number': '1'}
[2022-03-27, 18:29:27 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:28 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:29 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:30 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:31 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:32 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:33 UTC] {pod_manager.py:199} INFO - toto
[2022-03-27, 18:29:48 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:49 UTC] {pod_manager.py:214} WARNING - Pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 log read interrupted but container base still running
[2022-03-27, 18:29:50 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:50 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:50 UTC] {taskinstance.py:1278} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220326T000000, start_date=20220327T182926, end_date=20220327T182950
[2022-03-27, 18:29:50 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:29:50 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

joeyberkovitz commented 2 years ago

https://github.com/apache/airflow/pull/21694 looks good - tested with 2.2.5rc1 docker image.

conn.env:

TEST_CONN=mysql://username:password@host:1234/schema?param1=val1&param2=val2

airflow connections get TEST_CONN:

id   | conn_id   | conn_type | description | host | schema | login    | password | port | is_encrypted | is_extra_encrypted | extra_dejson                         | get_uri                                                           
=====+===========+===========+=============+======+========+==========+==========+======+==============+====================+======================================+===================================================================
None | TEST_CONN | mysql     | None        | host | schema | username | password | 1234 | False        | False              | {'param1': 'val1', 'param2': 'val2'} | mysql://username:password@host:1234/schema?param1=val1&param2=val2

frankcash commented 2 years ago

Just brought up a clean docker image with the RC to validate https://github.com/apache/airflow/pull/21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc1
typing_extensions                      4.1.1

Looks good to me

subkanthi commented 2 years ago

All good.

Checked #22241 - > classifier is there

21905 and #22527 were mostly to make sure the tests pass (they did :) )

22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":
    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

Tested #22027 on the tag 2.2.5rc1. Thanks

mike-mcdonald commented 2 years ago

Tested #20883 on 2.2.5rc2. Tests passed, using the connection form to test saving also worked as expected.

jkramer-ginkgo commented 2 years ago

21770 https://github.com/apache/airflow/pull/21770 confirmed with heavy

load

On Mon, Mar 28, 2022 at 12:06 PM Kanthi @.***> wrote:

All good.

Checked #22241 https://protect-us.mimecast.com/s/MBO2CkR0GkiZorph9lxCZ?domain=github.com

classifier is there

[image: image] https://protect-us.mimecast.com/s/T-b-C0Rr5NilYMLTwQ0Xm?domain=user-images.githubusercontent.com

21905

https://protect-us.mimecast.com/s/EAlGClY9JlUD6AvU1kMlB?domain=github.com and #22527 https://protect-us.mimecast.com/s/nf91CmZ9KmSGkArTDzQ7N?domain=github.com were mostly to make sure the tests pass (they did :) )

22027

https://protect-us.mimecast.com/s/OejUCn59L0fQrKYIp_80p?domain=github.com

I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi https://protect-us.mimecast.com/s/Xqz0Co2gMoUJ6B0c7JApt?domain=github.com if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json https://protect-us.mimecast.com/s/HRcMCgJ9AOfn4m7uo3y99?domain=pypi.org | jq ".info" | grep "extra == 'dask'":
"cloudpickle (>=1.4.1) ; extra == 'dask'",
"dask (>=2.9.0) ; extra == 'dask'",
"distributed (>=2.11.1) ; extra == 'dask'",
Tested #22027 https://protect-us.mimecast.com/s/OejUCn59L0fQrKYIp_80p?domain=github.com on the tag 2.2.5rc1. Thanks

— Reply to this email directly, view it on GitHub https://protect-us.mimecast.com/s/Fb_bCjR9EjikZ3vT5RsRD?domain=github.com, or unsubscribe https://protect-us.mimecast.com/s/GaXnCpYjNpUPpvBUx8OSR?domain=github.com . You are receiving this because you were mentioned.Message ID: @.***>

frankcash commented 2 years ago

Validated https://github.com/apache/airflow/pull/21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc2
typing_extensions                      4.1.1

potiuk commented 2 years ago

All good with RC3 from my point of view - including updated dependencies.

josh-fell commented 2 years ago

Verified #21192. Thanks @ephraimbuddy for organizing this release!

eladkal commented 2 years ago

Closing since 2.2.5 was released.

apache / airflow

Status of testing of Apache Airflow 2.2.5rc3 #22549

Body

Committer

21905 and #22527 were mostly to make sure the tests pass (they did :) )

22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

21905 and #22527 were mostly to make sure the tests pass (they did :) )

22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

21770 https://github.com/apache/airflow/pull/21770 confirmed with heavy

21905

22027