search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 953 forks source link

Bug: Jdbc_to_PubSub template missing #479

Closed sdhays closed 1 year ago

sdhays commented 1 year ago

Related Template(s)

Jdbc_to_PubSub

What happened?

The Jdbc_to_PubSub template isn't showing up in any folders in the gs://dataflow-templates/ bucket even though it is mentioned and linked in https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#java-database-connectivity-jdbc-to-pubsub. It seems it got left out of the deployment code.

It's also not listed in the README.

Beam Version

2.35.0

Relevant log output

No response

sdhays commented 1 year ago

Actually, it seems that the only googlecloud-to-googlecloud template that's getting built is JdbcToBigQuery. I don't see the others either, and when I build the project locally, only the Jdbc_To_BigQuery template gets built.

bvolpato commented 1 year ago

Check under the flex/ folder. Templates at v2 are Flex templates and generated differently:

gs://dataflow-templates/latest/flex/BigQuery_to_Elasticsearch gs://dataflow-templates/latest/flex/BigQuery_to_MongoDB gs://dataflow-templates/latest/flex/BigQuery_to_Parquet gs://dataflow-templates/latest/flex/Cdc_To_BigQuery_Template gs://dataflow-templates/latest/flex/Cloud_Datastream_to_BigQuery gs://dataflow-templates/latest/flex/Cloud_Datastream_to_SQL gs://dataflow-templates/latest/flex/Cloud_Datastream_to_Spanner gs://dataflow-templates/latest/flex/Cloud_PubSub_to_GCS_Text_Flex gs://dataflow-templates/latest/flex/Cloud_PubSub_to_MongoDB gs://dataflow-templates/latest/flex/Dataplex_BigQuery_to_GCS gs://dataflow-templates/latest/flex/Dataplex_File_Format_Conversion gs://dataflow-templates/latest/flex/Dataplex_JDBC_Ingestion gs://dataflow-templates/latest/flex/File_Format_Conversion gs://dataflow-templates/latest/flex/GCS_to_Elasticsearch gs://dataflow-templates/latest/flex/Jdbc_to_BigQuery_Flex gs://dataflow-templates/latest/flex/Jdbc_to_PubSub gs://dataflow-templates/latest/flex/Kafka_to_BigQuery gs://dataflow-templates/latest/flex/MongoDB_to_BigQuery gs://dataflow-templates/latest/flex/MongoDB_to_BigQuery_CDC gs://dataflow-templates/latest/flex/PubSub_Avro_to_BigQuery gs://dataflow-templates/latest/flex/PubSub_CDC_to_BigQuery gs://dataflow-templates/latest/flex/PubSub_Proto_to_BigQuery gs://dataflow-templates/latest/flex/PubSub_to_Elasticsearch gs://dataflow-templates/latest/flex/Pubsub_to_Jdbc gs://dataflow-templates/latest/flex/Spanner_Change_Streams_to_BigQuery gs://dataflow-templates/latest/flex/Spanner_Change_Streams_to_Google_Cloud_Storage gs://dataflow-templates/latest/flex/Streaming_Data_Generator

sdhays commented 1 year ago

Thanks, @bvolpato! I missed that.