search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.65k stars 14.18k forks source link

Status of testing Providers that were prepared on July 27, 2021 (rc2) #17268

Closed potiuk closed 3 years ago

potiuk commented 3 years ago

This is the second attempt (rc2) of releasing July Providers :). I carried over the "checkmarks" from the last testing (#17037) so there is no need to re-test those.

I have a kind request for all the contributors to the latest provider packages release. Could you help us to test the RC versions of the providers and let us know in the comment, if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 2.1.0rc2

New Providers

raphaelauv commented 3 years ago

for docker 2.1.0rc2 it's all good

docker in docker work and also the XCOM

t3 = DockerOperator(
    api_version='auto',
    docker_url="unix://var/run/docker.sock",
    command='/bin/echo toto',
    image='centos:latest',
    network_mode='bridge',
    do_xcom_push=True,
    task_id='docker_op_tester',
    dag=dag,
)

give 

[2021-07-27 19:38:44,647] {docker.py:246} INFO - Starting docker container from image centos:latest
[2021-07-27 19:38:44,682] {docker.py:256} WARNING - Using remote engine or docker-in-docker and mounting temporary volume from host is not supported. Falling back to `mount_tmp_dir=False` mode. You can set `mount_tmp_dir` parameter to False to disable mounting and remove the warning
[2021-07-27 19:38:46,975] {docker.py:307} INFO - toto
[2021-07-27 19:38:47,664] {taskinstance.py:1204} INFO - Marking task as SUCCESS. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210727T150000, start_date=20210727T193839, end_date=20210727T193847
[2021-07-27 19:38:47,770] {taskinstance.py:1265} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-07-27 19:38:47,815] {local_task_job.py:149} INFO - Task exited with return code 0

and 

t3 = DockerOperator(
    api_version='auto',
    docker_url="unix://var/run/docker.sock",
    command='/bin/echo toto',
    image='centos:latest',
    network_mode='bridge',
    mount_tmp_dir=False,
    do_xcom_push=True,
    task_id='docker_op_tester',
    dag=dag,
)

give

[2021-07-27 19:39:27,176] {docker.py:246} INFO - Starting docker container from image centos:latest
[2021-07-27 19:39:28,091] {docker.py:307} INFO - toto
[2021-07-27 19:39:28,383] {taskinstance.py:1204} INFO - Marking task as SUCCESS. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210727T180000, start_date=20210727T193926, end_date=20210727T193928
[2021-07-27 19:39:28,424] {taskinstance.py:1265} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-07-27 19:39:28,464] {local_task_job.py:149} INFO - Task exited with return code 0

and in all case a XCOM

key : return_value value : toto

levyitay commented 3 years ago

qubole: 2.0.0rc2 tested and working

josh-fell commented 3 years ago

Contributions for Google, Amazon, and Salesforce providers look good!

pmalafosse commented 3 years ago

amazon: 2.1.0rc2:

I just tried it and ECSOperator is working as expected

pmalafosse commented 3 years ago

This one is fine too:

Goodkat commented 3 years ago

Tested again the odbc connection, it works as expected with the boolean parameters: https://github.com/apache/airflow/issues/15016#issuecomment-826650629

potiuk commented 3 years ago

Getting very good progress on testing with this release :) . Thanks a bunch for those! looking forward for more quick tests :)

pavelhlushchanka commented 3 years ago

16820 is also good to go

baolsen commented 3 years ago

16771 and #16612 tested and working as expected.

On Wed, 28 Jul 2021, 10:41 AM Pavel Hlushchanka, @.***> wrote:

16820 https://github.com/apache/airflow/pull/16820 is also good to go

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/17268#issuecomment-888127323, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLEIYZHSJOW72BLXQVJWYTTZ67CFANCNFSM5BCTM7EQ .

ayushchauhan0811 commented 3 years ago

17081 is also good to go

iostreamdoth commented 3 years ago

fix: get_cluster call project_id as self.project_id (#17075) looks good.

infused-kim commented 3 years ago

Google ads #17160 is working fine, but I am the one who submitted the PR and it would be good to get it tested by a few more people.

potiuk commented 3 years ago

Google ads #17160 is working fine, but I am the one who submitted the PR and it would be good to get it tested by a few more people.

@overbryd and @rv-irosario - since this was addressing your issues #17113 and #17111 - could you please also take a look if https://pypi.org/project/apache-airflow-providers-google/5.0.0rc2 solves the ads compatibility ?

potiuk commented 3 years ago

Hey @oyarushe @scottypate @uranusjr @msumit @Daniel-Han-Yang @rv-irosario @aslantar @malthe @LukeHong @sunkickr @mmenarguezpear @fredthomsen @ngaranko @namjals @dstandish @BKronenbitter @freget @ashb @ciancolo @samgans - last chance to test before we release later tonight :)

oyarushe commented 3 years ago

I'm trying to check #16521, but the fix was provided in DbApiHook, which is a general class and I suppose will be released with Airflow itself. So I think we need to release the Postgres provider after Airflow.

potiuk commented 3 years ago

So I think we need to release the Postgres provider after Airflow.

Ah yeah. This is backwards-compatible change, so it's ok to release Postgres now, but indeed testing will be possible only with new Airflow released (and it i's a new feature so it should be only released in 2.2. Good that you raised it!

ciancolo commented 3 years ago

Tested #16365 and #17125 on my local Tableau Server and they works.

potiuk commented 3 years ago

Thanks everyone who tested! This is really great help for the community. The vote has passed and I am proceeding to release the providers!