search

GoogleCloudDataproc / spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.
Apache License 2.0
358 stars 189 forks source link

Viewing connector metrics #1252

Closed Swebask closed 5 days ago

Swebask commented 5 days ago

It isn't very clear from the documentation where the connector metrics can be viewed. I'm running a job on a Dataproc 2.1 cluster using connector version gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.30.0.jar.

davidrabinowitz commented 5 days ago

You can see them in the Spark UI. Please upgrade to version 0.39.1 first as described here

Swebask commented 5 days ago

Thank you for your quick response. Upgraded to version 0.39.1, still don't see it. Is there a particular section of the Spark UI to look for the metrics?

isha97 commented 4 days ago

@Swebask After you go to the applicate page in the Spark UI, navigate to SQL/Dataframe section from the top. Then you can select the job and get the job graph which will show the metrics. (https://spark.apache.org/docs/3.0.0/web-ui.html#sql-tab)