Migrate Google example DAGs to new design

[potiuk]

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Google provider. It is created to track progress of their migration.

List of paths to example DAGs:

• [x] airflow/providers/google/leveldb/example_dags/example_leveldb.py
• [x] airflow/providers/google/firebase/example_dags/example_firestore.py
• [x] airflow/providers/google/ads/example_dags/example_ads.py
• [x] airflow/providers/google/suite/example_dags/example_sql_to_sheets.py
• [x] airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py
• [x] airflow/providers/google/suite/example_dags/example_gcs_to_gdrive.py
• [x] airflow/providers/google/suite/example_dags/example_sheets.py
• [x] airflow/providers/google/marketing_platform/example_dags/example_analytics.py
• [ ] airflow/providers/google/marketing_platform/example_dags/example_display_video.py
• [x] airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py
• [x] airflow/providers/google/marketing_platform/example_dags/example_search_ads.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs_to_sftp.py
• [x] airflow/providers/google/cloud/example_dags/example_oracle_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_compute_igm.py
• [x] airflow/providers/google/cloud/example_dags/example_automl_vision_classification.py
• [x] airflow/providers/google/cloud/example_dags/example_datacatalog.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_mssql.py
• [x] airflow/providers/google/cloud/example_dags/example_mssql_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_mysql_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
• [x] airflow/providers/google/cloud/example_dags/example_natural_language.py
• [x] airflow/providers/google/cloud/example_dags/example_life_sciences.py
• [x] airflow/providers/google/cloud/example_dags/example_automl_nl_text_extraction.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_classification.py
• [x] airflow/providers/google/cloud/example_dags/example_trino_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_transfer.py
• [x] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
• [x] airflow/providers/google/cloud/example_dags/example_translate.py
• [x] airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_compute_ssh.py
• [ ] airflow/providers/google/cloud/example_dags/example_dataflow_sql.py
• [x] airflow/providers/google/cloud/example_dags/example_text_to_speech.py
• [ ] airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
• [ ] airflow/providers/google/cloud/example_dags/example_looker.py
• [x] airflow/providers/google/cloud/example_dags/example_sftp_to_gcs.py
• [ ] airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs_to_bigquery.py
• [ ] airflow/providers/google/cloud/example_dags/example_postgres_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_spanner.py
• [x] airflow/providers/google/cloud/example_dags/example_gdrive_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_dlp.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_dts.py
• [x] airflow/providers/google/cloud/example_dags/example_datastore.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_classification.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_vision_object_detection.py
• [x] airflow/providers/google/cloud/example_dags/example_bigtable.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
• [ ] airflow/providers/google/cloud/example_dags/example_vertex_ai.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_tracking.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
• [x] airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
• [ ] airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_functions.py
• [x] airflow/providers/google/cloud/example_dags/example_compute.py
• [x] airflow/providers/google/cloud/example_dags/example_sheets_to_gcs.py
• [ ] airflow/providers/google/cloud/example_dags/example_dataflow.py
• [x] airflow/providers/google/cloud/example_dags/example_datafusion.py
• [ ] airflow/providers/google/cloud/example_dags/example_cloud_sql_query.py
• [x] airflow/providers/google/cloud/example_dags/example_translate_speech.py
• [x] airflow/providers/google/cloud/example_dags/example_automl_tables.py
• [x] airflow/providers/google/cloud/example_dags/example_cloud_memorystore.py
• [x] airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_mlengine.py
• [x] airflow/providers/google/cloud/example_dags/example_dataprep.py
• [ ] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
• [x] airflow/providers/google/cloud/example_dags/example_pubsub.py
• [x] airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_s3_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_gdrive_to_local.py
• [x] airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_workflows.py
• [x] airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py
• [x] airflow/providers/google/cloud/example_dags/example_dataproc.py
• [x] airflow/providers/google/cloud/example_dags/example_speech_to_text.py
• [x] airflow/providers/google/cloud/example_dags/example_cloud_composer.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_translation.py
• [x] airflow/providers/google/cloud/example_dags/example_stackdriver.py
• [x] airflow/providers/google/cloud/example_dags/example_tasks.py
• [x] airflow/providers/google/cloud/example_dags/example_calendar_to_gcs.py
• [x] airflow/providers/google/cloud/example_dags/example_video_intelligence.py
• [x] airflow/providers/google/cloud/example_dags/example_cloud_build.py
• [ ] airflow/providers/google/cloud/example_dags/example_dataflow_flex_template.py
• [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_sentiment.py
• [x] airflow/providers/google/cloud/example_dags/example_cloud_sql.py
• [x] airflow/providers/google/cloud/example_dags/example_vision.py

In progress:

• airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
• airflow/providers/google/cloud/example_dags/example_dataflow.py
• airflow/providers/google/cloud/example_dags/example_datafusion.py
• airflow/providers/google/cloud/example_dags/example_dataproc.py
• airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
• airflow/providers/google/cloud/example_dags/example_datastore.py
• airflow/providers/google/cloud/example_dags/example_gcs.py
• airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
• airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
• airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
• airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

[mnojek]

Hey, please update the checklist.

These example DAGs are being removed by PR #22311:

• airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
• airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
• airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py

And these are WIP (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication):

• airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
• airflow/providers/google/cloud/example_dags/example_dataflow.py
• airflow/providers/google/cloud/example_dags/example_datafusion.py
• airflow/providers/google/cloud/example_dags/example_dataproc.py
• airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
• airflow/providers/google/cloud/example_dags/example_datastore.py
• airflow/providers/google/cloud/example_dags/example_gcs.py
• airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
• airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
• airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
• airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

[potiuk]

Updated

[potiuk]

BTW. I will automate refreshing the list shortly :) @bhirsz :)

[chethanuk]

Hi @potiuk Can we update the checklist if and when possible [So Can know which needs to be done, which are WIP]. Thanks :)

[potiuk]

Sure. I merge all the green ones and re-run the script to update the checklists

[eladkal]

Do we still need this tracking issue?