search

GoogleCloudPlatform / opentelemetry-operations-java

Apache License 2.0
71 stars 41 forks source link

<Question> Do we have Auto instrumentation support for Cloud composer ? #324

Closed Santosh-Kumar-Perumal closed 4 months ago

psx95 commented 4 months ago

Hi Santosh, I have not used Cloud Composer before, but looking around it seems like you can use various Google Cloud Operators to schedule work on Cloud Composer.

I think it should be possible to instrument individual tasks within the DAG. For instance, the Google Cloud Functions Operator can be used to deploy a Cloud Function from a source. I imagine if the Cloud Function is instrumented with OpenTelemetry, you would get the telemetry from it.

Using AutoInstrumentation would depend on if it is supported by the environment itself. For instance, you can run OTel Java agent in a Cloud Run container, so you can probably use auto-instrumentation when using Google Cloud Run Operator in the Cloud Composer.

If you're asking about instrumenting the entire Cloud Composer workflow itself - after looking around, it seems like Apache Airflow (Cloud Composer is built on top of it) produces some metrics by default and you can use OTel Collector to collect those metrics, but I don't think this would be called "Auto-instrumenting Cloud Composer".

Santosh-Kumar-Perumal commented 4 months ago

Thanks @psx95 for the details much appreciated.