search

GoogleCloudPlatform / DataflowJavaSDK

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
http://cloud.google.com/dataflow
855 stars 324 forks source link

v2.1.0 streaming mode missing metrics and basic diagnostics #602

Closed okorz001 closed 6 years ago

okorz001 commented 6 years ago

After upgrading form v2.0.0 to v2.1.0 today, I was disappointed to discover not only are streaming metrics still missing in our dataflow job view, but we have lost basic step diagnostics such as "Elements added", "Estimated size" and "Wall time". The only indicator that the entire pipeline is even running is that the global resource metrics like "Total vCPU time" are still increasing.

Is there some other app change required beyond increasing the com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all dependency version?

Metrics are created with org.apache.beam.sdk.metrics.Metrics static factories (counter, gauge).

okorz001 commented 6 years ago

Metrics are working now.

I noticed that the job page started complaining that we were still running 2.0.0, so I gave 2.1.0 another shot today. I am not sure if anything changed on the cloud side.