Closed jayadeep-jayaraman closed 5 years ago
You might have to save to somewhere after the last .map
? Otherwise the runner might ignore dummy transforms that don't end up PDone
.
Thanks @nevillelyh . If i remove the pubsub source and add some dummy data inline then the final output gets printed. Do you think saving it somewhere might be related to a streaming pipeline?
Hi @nevillelyh - I added the below code to write to Bigquery but still the job is not finishing and the table is not getting created in BQ.
consolidated_view.map(kv => TableRow("orderId" -> kv.orderId, "pickup_location" -> kv.pickup_location,"dropoff_location"->kv.dropoff_location,"resturantId"->kv.resturantId))
.saveAsBigQuery("jayadeep-etl-platform:demo.orders", schema, WRITE_APPEND, CREATE_IF_NEEDED)
The issue was with Direct Runner, when i changed it to Dataflow runner the issue got resolved.
Hi,
I am building a simple prototype wherein I am reading data from Pubsub and using BeamSQL, code snippet as below
I am using Directrunner for testing it locally and I am able to see the results right until the beam sql is executed. The output from beam sql is not getting printed.