search

GoogleCloudPlatform / DataflowJavaSDK

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
http://cloud.google.com/dataflow
855 stars 324 forks source link

Cloud Dataflow Template #579

Closed kpr6 closed 3 years ago

kpr6 commented 7 years ago

I made a dataflow template to load .csv files from bucket to BigQuery tables using TemplatingDataflowPipelineRunner, the first time when I run a template job after it's creation, it works fine and loads data to respective location. But when I re-use the template to run another job to migrate another file, the job succeeds but data isn't moved which is strange. Is it that template can only be used once?

HayoVanLoon commented 7 years ago

Batch jobs like the one you describe and BigQueryIO do not (yet) play well together. See #550

tade82 commented 5 years ago

The same issue exists when you read data from BigQuery and write to Pub/Sub

repl-abhishek-srivastava commented 4 years ago

Hi All, +1 I am sending data from bigquery to datastore. It works fine for the first time then for each execution it is successfull on UI , but no process is actually done.