Can't export telemetry data from my app engine to gcp monitoring

[owensulei3841]

Hi everyone, I try to create 2 yaml file in my repo called collector-config-gcp.yaml and docker-compose-gcp.yaml, just want to export metrcis from app engine to aws and gcp console. Right now I can't see any telemetry data from my console. Since I'm using the openTelemetry exporter not the gcp exporter. I know if I use the gcp exporter to do the configuration, it will works. But I just want to ask, if I can create a single Collector using yaml file, and that collector can export data to aws, gcp and etc ? This is my repo : https://github.com/owensulei3841/otlp

Thank you

[psx95]

So, if I am understanding correctly, you have a SpringBootApplication and you want to export the telemetry data to both AWS and GCP. Looking at your config's exporter section, I see that you have configured AWS exporter but Google Cloud exporter configuration is missing.

I think it should be possible if you add config for googlecloud as well. You can refer to Google Cloud Exporter in the opentelemetry-collector-contrib repository.

[owensulei3841]

Hi Paranv, In this case, can I use openTelemetry exporter only ? Or I have to change the exporter to GCP exporter?

On Tue, Apr 18, 2023 at 11:34 AM Pranav Sharma @.***> wrote:

So, if I am understanding correctly, you have a SpringBootApplication and you want to export the telemetry data to both AWS and GCP. Looking at your config's exporter https://github.com/owensulei3841/otlp/blob/ab867cd108797c9e4a6b7c91021846cd69646b71/collector-config-gcp.yaml#L12 section, I see that you have configured AWS exporter but Google Cloud exporter configuration is missing.

I think it should be possible if you add config for googlecloud as well. You can refer to Google Cloud Exporter https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter in the opentelemetry-collector-contrib repository.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1513627260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUC367557QT3LJJY5HTXB3NDPANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1513627260@ github.com>

[owensulei3841]

And by the way, currently I'm using an app engine for my service, do I need to install docker and container in my app engine in order to let my collector run? Thank you

Kevin

On Tue, Apr 18, 2023 at 12:06 PM kevin su @.***> wrote:

Hi Paranv, In this case, can I use openTelemetry exporter only ? Or I have to change the exporter to GCP exporter?

On Tue, Apr 18, 2023 at 11:34 AM Pranav Sharma @.***> wrote:

So, if I am understanding correctly, you have a SpringBootApplication and you want to export the telemetry data to both AWS and GCP. Looking at your config's exporter https://github.com/owensulei3841/otlp/blob/ab867cd108797c9e4a6b7c91021846cd69646b71/collector-config-gcp.yaml#L12 section, I see that you have configured AWS exporter but Google Cloud exporter configuration is missing.

I think it should be possible if you add config for googlecloud as well. You can refer to Google Cloud Exporter https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter in the opentelemetry-collector-contrib repository.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1513627260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUC367557QT3LJJY5HTXB3NDPANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1513627260@ github.com>

[psx95]

Hi Paranv, In this case, can I use openTelemetry exporter only ? Or I have to change the exporter to GCP exporter?

No, I believe you will have to make changes to your collector-config-yaml to add configuration for googlecloud exporter to view telemetry data in Google Cloud Console.

[psx95]

And by the way, currently I'm using an app engine for my service, do I need to install docker and container in my app engine in order to let my collector run? Thank you Kevin

I think you can run the collector binary directly as well, without containerizing it as well, but I have not tried this method before.

The App Engine documentation here mentions that for Java 11/17 applications, it runs your application on a Ubuntu Linux base image. So you might be able to run the collector binary for linux directly.

[owensulei3841]

Hi Pranav, Can I ask how to run the collector? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter#prerequisite-authenticating [image: Screenshot 2023-04-18 at 3.29.59 PM.png] I tried to open my gcp terminal and run it. It seems not the correct location. [image: Screenshot 2023-04-18 at 3.28.26 PM.png]

On Tue, Apr 18, 2023 at 12:42 PM Pranav Sharma @.***> wrote:

And by the way, currently I'm using an app engine for my service, do I need to install docker and container in my app engine in order to let my collector run? Thank you Kevin

I think you can run the collector binary directly as well, without containerizing it as well, but I have not tried this method before.

The App Engine documentation here https://cloud.google.com/appengine/docs/standard/java-gen2/runtime#the_runtimes_open_jdk_environment mentions that for Java 11/17 applications, it runs your application on a Ubuntu Linux base image. So you might be able to run the collector binary for linux directly.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1513704553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUDBDP6TZMQZBGZ2P4LXB3VCZANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1513704553@ github.com>

[owensulei3841]

By the way, right now I can see the metrics names from the monitoring dashboard, but when I click each individual metrics, it shows no data like following, does it mean my collector and gcp monitoring are connected, but since I'm missing some permissions, there is no data shown in the panel. The metrics names: [image: Screenshot 2023-04-18 at 4.08.09 PM.png]

when select a single metrics: [image: Screenshot 2023-04-18 at 4.11.24 PM.png]

This is my permission list for app engine: [image: Screenshot 2023-04-18 at 4.12.52 PM.png]

On Tue, Apr 18, 2023 at 3:30 PM kevin su @.***> wrote:

Hi Pranav, Can I ask how to run the collector? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter#prerequisite-authenticating [image: Screenshot 2023-04-18 at 3.29.59 PM.png] I tried to open my gcp terminal and run it. It seems not the correct location. [image: Screenshot 2023-04-18 at 3.28.26 PM.png]

On Tue, Apr 18, 2023 at 12:42 PM Pranav Sharma @.***> wrote:

And by the way, currently I'm using an app engine for my service, do I need to install docker and container in my app engine in order to let my collector run? Thank you Kevin

I think you can run the collector binary directly as well, without containerizing it as well, but I have not tried this method before.

The App Engine documentation here https://cloud.google.com/appengine/docs/standard/java-gen2/runtime#the_runtimes_open_jdk_environment mentions that for Java 11/17 applications, it runs your application on a Ubuntu Linux base image. So you might be able to run the collector binary for linux directly.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1513704553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUDBDP6TZMQZBGZ2P4LXB3VCZANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1513704553@ github.com>

[psx95]

It seems like the images are not attached correctly - I see that you are replying by email. To upload images I think you will need to do it via Github's website.

[owensulei3841]

Sorry about the image, I just attach them below again. Right now I can see the metrics names from the monitoring dashboard, but when I click each individual metrics, it shows no data like following, does it mean my collector and gcp monitoring are connected, but since I'm missing some permissions, there is no data shown in the panel. The metrics names:

when select a single metrics:

This is my permission list for app engine: I'm using the default permission for app engine

[psx95]

No, if the metric names appears here, that means it was successfully pushed to cloud monitoring at least once. Also I see that "Show only active metrics" switch is enabled, which means that these metrics were seen in the last 25 hours. Can you try increasing the time range from "Last 6 hours" to maybe Last 1 or 2 days ?

[owensulei3841]

Somehow I can't see the metrics names anymore. Since it's really hard to debug on app engine, so I'm trying to debug in my local environment. Sorry for the confusion. Those are the steps I am doing in my local:

1. I download otelcol and otelcol-contrib file into my project
2. I download docker by docker pull otel/opentelemetry-collector-contrib
3. I signup my gcp account through using gcloud auth application-default login
4. I compile my spring project and run it in local
5. I run ./otelcol-contrib --config=config.yaml This is my current config.yaml file

This is my log after running the ./otelcol-contrib --config=config.yaml

2023-04-19T13:44:14.686-0700 info service/telemetry.go:90 Setting up own telemetry... 2023-04-19T13:44:14.687-0700 info service/telemetry.go:116 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} 2023-04-19T13:44:14.695-0700 info memorylimiterprocessor@v0.75.0/memorylimiter.go:149 Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "total_memory_mib": 32768, "limit_percentage": 65, "spike_limit_percentage": 20} 2023-04-19T13:44:14.695-0700 info memorylimiterprocessor@v0.75.0/memorylimiter.go:113 Memory limiter configured {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "limit_mib": 21299, "spike_limit_mib": 6553, "check_interval": 1} 2023-04-19T13:44:14.695-0700 info service/service.go:129 Starting otelcol-contrib... {"Version": "0.75.0", "NumCPU": 10} 2023-04-19T13:44:14.695-0700 info extensions/extensions.go:41 Starting extensions... 2023-04-19T13:44:14.695-0700 warn internal/warning.go:51 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2023-04-19T13:44:14.695-0700 info otlpreceiver@v0.75.0/otlp.go:94 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"} 2023-04-19T13:44:14.696-0700 warn internal/warning.go:51 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2023-04-19T13:44:14.696-0700 info otlpreceiver@v0.75.0/otlp.go:112 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"} 2023-04-19T13:44:14.696-0700 info service/service.go:146 Everything is ready. Begin running and processing data.

This is when i calling that service: 2023-04-19T13:44:50.239-0700 error exporterhelper/queued_retry.go:367 Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "failed to export time series to GCM: rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)."} go.opentelemetry.io/collector/exporter/exporterhelper.(retrySender).send go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/queued_retry.go:367 go.opentelemetry.io/collector/exporter/exporterhelper.(metricsSenderWithObservability).send go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/metrics.go:136 go.opentelemetry.io/collector/exporter/exporterhelper.(queuedRetrySender).start.func1 go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/queued_retry.go:205 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/internal/bounded_memory_queue.go:58 2023-04-19T13:44:50.239-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"otlp.exporter.exported\" type:\"workload.googleapis.com/otlp.exporter.exported\" labels:{key:\"service_name\"} labels:{key:\"instrumentation_version\"} labels:{key:\"instrumentation_source\"} labels:{key:\"success\"} labels:{key:\"type\"} metric_kind:CUMULATIVE value_type:INT64 display_name:\"otlp.exporter.exported\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023/04/19 13:44:50 failed to export to Google Cloud Trace: rpc error: code = PermissionDenied desc = The caller does not have permission 2023-04-19T13:44:50.285-0700 error exporterhelper/queued_retry.go:367 Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "traces", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = The caller does not have permission"} go.opentelemetry.io/collector/exporter/exporterhelper.(retrySender).send go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/queued_retry.go:367 go.opentelemetry.io/collector/exporter/exporterhelper.(tracesExporterWithObservability).send go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/traces.go:137 go.opentelemetry.io/collector/exporter/exporterhelper.(queuedRetrySender).start.func1 go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/queued_retry.go:205 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/collector/exporter@v0.75.0/exporterhelper/internal/bounded_memory_queue.go:58 2023-04-19T13:44:50.422-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"otlp.exporter.seen\" type:\"workload.googleapis.com/otlp.exporter.seen\" labels:{key:\"instrumentation_version\"} labels:{key:\"instrumentation_source\"} labels:{key:\"service_name\"} labels:{key:\"type\"} metric_kind:CUMULATIVE value_type:INT64 display_name:\"otlp.exporter.seen\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:50.645-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"http.server.duration\" type:\"workload.googleapis.com/http.server.duration\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"http_flavor\"} labels:{key:\"http_method\"} labels:{key:\"http_route\"} labels:{key:\"http_scheme\"} labels:{key:\"http_status_code\"} labels:{key:\"net_host_name\"} labels:{key:\"net_host_port\"} metric_kind:CUMULATIVE value_type:DISTRIBUTION unit:\"ms\" description:\"The duration of the inbound HTTP request\" display_name:\"http.server.duration\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:50.798-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"http.server.active_requests\" type:\"workload.googleapis.com/http.server.active_requests\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"http_flavor\"} labels:{key:\"http_method\"} labels:{key:\"http_scheme\"} labels:{key:\"net_host_name\"} metric_kind:GAUGE value_type:INT64 unit:\"requests\" description:\"The number of concurrent HTTP requests that are currently in-flight\" display_name:\"http.server.active_requests\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.055-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"http.client.duration\" type:\"workload.googleapis.com/http.client.duration\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"http_method\"} labels:{key:\"http_status_code\"} labels:{key:\"net_peer_name\"} labels:{key:\"net_peer_port\"} metric_kind:CUMULATIVE value_type:DISTRIBUTION unit:\"ms\" description:\"The duration of the outbound HTTP request\" display_name:\"http.client.duration\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.262-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"processedSpans\" type:\"workload.googleapis.com/processedSpans\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"dropped\"} labels:{key:\"spanProcessorType\"} metric_kind:CUMULATIVE value_type:INT64 unit:\"1\" description:\"The number of spans processed by the BatchSpanProcessor. [dropped=true if they were dropped due to high throughput]\" display_name:\"processedSpans\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.409-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"queueSize\" type:\"workload.googleapis.com/queueSize\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"spanProcessorType\"} metric_kind:GAUGE value_type:INT64 unit:\"1\" description:\"The number of spans queued\" display_name:\"queueSize\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.619-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"process.runtime.jvm.system.cpu.utilization\" type:\"workload.googleapis.com/process.runtime.jvm.system.cpu.utilization\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:GAUGE value_type:DOUBLE unit:\"1\" description:\"Recent cpu utilization for the whole system\" display_name:\"process.runtime.jvm.system.cpu.utilization\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.814-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"process.runtime.jvm.cpu.utilization\" type:\"workload.googleapis.com/process.runtime.jvm.cpu.utilization\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:GAUGE value_type:DOUBLE unit:\"1\" description:\"Recent cpu utilization for the process\" display_name:\"process.runtime.jvm.cpu.utilization\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:51.988-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"process.runtime.jvm.classes.loaded\" type:\"workload.googleapis.com/process.runtime.jvm.classes.loaded\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:CUMULATIVE value_type:INT64 unit:\"1\" description:\"Number of classes loaded since JVM start\" display_name:\"process.runtime.jvm.classes.loaded\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:52.216-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"process.runtime.jvm.memory.init\" type:\"workload.googleapis.com/process.runtime.jvm.memory.init\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"pool\"} labels:{key:\"type\"} metric_kind:GAUGE value_type:INT64 unit:\"By\" description:\"Measure of initial memory requested\" display_name:\"process.runtime.jvm.memory.init\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023/04/19 13:44:54 failed to export to Google Cloud Trace: rpc error: code = PermissionDenied desc = The caller does not have permission 2023-04-19T13:44:54.909-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.api.nestedSpan1.latency\" type:\"workload.googleapis.com/custom.metric.api.nestedSpan1.latency\" labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"instrumentation_source\"} metric_kind:CUMULATIVE value_type:DISTRIBUTION description:\"This is nestedSpan1 api latency.\" display_name:\"custom.metric.api.nestedSpan1.latency\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:55.111-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.heap.memory\" type:\"workload.googleapis.com/custom.metric.heap.memory\" labels:{key:\"service_name\"} labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} metric_kind:GAUGE value_type:DOUBLE unit:\"byte\" description:\"Reports heap memory utilization.\" display_name:\"custom.metric.heap.memory\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:55.311-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.number.of.exec\" type:\"workload.googleapis.com/custom.metric.number.of.exec\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:CUMULATIVE value_type:INT64 unit:\"int\" description:\"Count the number of executions.\" display_name:\"custom.metric.number.of.exec\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:55.513-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.number.of.up.and.down.counter\" type:\"workload.googleapis.com/custom.metric.number.of.up.and.down.counter\" labels:{key:\"service_name\"} labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"api_url\"} metric_kind:CUMULATIVE value_type:INT64 unit:\"int\" description:\"Number of upAndDown counter.\" display_name:\"custom.metric.number.of.up.and.down.counter\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:55.711-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.number.of.request\" type:\"workload.googleapis.com/custom.metric.number.of.request\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} labels:{key:\"request_name\"} metric_kind:CUMULATIVE value_type:INT64 unit:\"int\" description:\"Count the number of requests.\" display_name:\"custom.metric.number.of.request\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:55.912-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.api.simpleSpan.latency\" type:\"workload.googleapis.com/custom.metric.api.simpleSpan.latency\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:CUMULATIVE value_type:DISTRIBUTION description:\"This is simpleSpan api latency.\" display_name:\"custom.metric.api.simpleSpan.latency\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:56.112-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.number.of.async.counter\" type:\"workload.googleapis.com/custom.metric.number.of.async.counter\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:CUMULATIVE value_type:INT64 description:\"Number of async counter.\" display_name:\"custom.metric.number.of.async.counter\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338 2023-04-19T13:44:56.313-0700 error collector@v0.36.0/metrics.go:362 Unable to send metric descriptor. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).", "metric_descriptor": "name:\"custom.metric.number.of.async.up.and.down.counter\" type:\"workload.googleapis.com/custom.metric.number.of.async.up.and.down.counter\" labels:{key:\"instrumentation_source\"} labels:{key:\"instrumentation_version\"} labels:{key:\"service_name\"} metric_kind:GAUGE value_type:INT64 description:\"Number of async upaAndDown counter.\" display_name:\"custom.metric.number.of.async.up.and.down.counter\""} github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptor github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:362 github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(MetricsExporter).exportMetricDescriptorRunner github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.36.0/metrics.go:338

[owensulei3841]

It seems i got permission denied from the previous comments, I am using kevinsu3841@gmail.com when auth the gcp in local, this is my permissions:

[owensulei3841]

By the way, if I remove the "# "project: local-test" from the config.yaml file, like shown in the picture

I will encounter error like Error: failed to build pipelines: failed to create "googlecloud" exporter for data type "metrics": no project set in config, or found with application default credentials 2023/04/19 14:23:41 collector server run finished with error: failed to build pipelines: failed to create "googlecloud" exporter for data type "metrics": no project set in config, or found with application default credentials

Does it mean I didn't setup my local credential correctly? Thank you

[psx95]

Regarding commenting out the project: local-test issue - That is expected, you are required to pass the project ID.

Looking at your screenshots, it seems like you have the owner permission which would contain all required permissions, so I don't think lacking permissions is an issue. (Just FYI, for production projects you should not use this permission)

I would check if the selected project ID matches what you have mentioned in the collector config here.

• You can run gcloud config get-value project this gives you the current active/selected project.
• If you used gcloud auth application-default login for authentication, can you check quota_project_id field in the credentials file ? - this should match the selected project.
  • On a MacOS/Linux machine you can find the file in $HOME/.config/gcloud/application_default_credentials.json.

[owensulei3841]

Hi Pranav, I’m really appreciated! it works in my local after I put project id into it. I thought it was optional. after I tested it in my local, I push all changes to gcp app engine. I download the binary, image into the app engine, I use project ssh to login and start the collector. The collector works but no data in console. Is this right way to day that? Or I should download and run the collector in each vm instance? Is there a best practice to do that? Thank you

Pranav Sharma @.***>于2023年4月20日 周四07:49写道：

Regarding commenting out the project: local-test issue - That is expected, you are required to pass the project ID.

Looking at your screenshots, it seems like you have the owner permission which would contain all required permissions, so I don't think lacking permissions is an issue. (Just FYI, for production projects you should not use this permission)

I would check if the selected project ID matches what you have mentioned in the collector config here.

• You can run gcloud config get-value project this gives you the current active/selected project.
• If you used gcloud auth application-default login for authentication, can you check quota_project_id field in the credentials file ? - this should match the selected project.
  • On a MacOS/Linux machine you can find the file in $HOME/.config/gcloud/application_default_credentials.json.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1516467572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUFH7O6LYBU7E23SV7DXCFEJJANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1516467572@ github.com>

[psx95]

I have not used collector with Google App Engine so I will need to check, but generally speaking (like for GCE) the collector process and the application need to run on the same VM so that they can communicate.

Does the 5th point in the Getting Started section of the Collector documentation help ? (Like verify if your application is sending metrics to the correct place from where the collector can scrape them)

[psx95]

I just checked the https://opentelemetry.io/docs/collector/configuration/#receivers section and it seems like its not necessarily required for Collector and your application to run on the same VM as long as the application is able to send data (metrics, logs, traces) to the correct endpoint/port on which the collector is configured to run.

[owensulei3841]

Hi Pranav, I still can't export data from app engine to monitoring/trace. I tried to log the server use project/vm ssh, on that default path. I download the binary files, and create a single config.yaml file there, then run the command ./otelcol-contrib --config=config.yaml This is the image when i run it in app engine:

I call this server by : curl -X POST https://alpine-eon-381904.uw.r.appspot.com/nestedSpan1 but no result Thank you

[psx95]

Just to confirm, do you get any response back for the post request you make ?

[owensulei3841]

I got the response, you can try to curl that app engine service. I think it's working. But no metrcs data export to gcp monitoring

[owensulei3841]

Is there another way to start my collector in app engine? Right now, it seems my app engine and collector are both running, but still no data export to dashboard.

[psx95]

I can try it on my end once, but could you list the steps you followed to deploy the service and collector on app engine ?

Also, are you using App Engine Flex or Standard environment? I looked at https://github.com/owensulei3841/otlp, I did not see an app.yaml - looking at app engine documentation, I think this file was required.

[owensulei3841]

You are correct I didn't use app.yaml file to deploy my project. Currently I'm just download the binary and put config.ymal file at the same folder, then run the collector. It seems doesn't work. Let me try the app engine method. Thank you

[owensulei3841]

Hi Paranv, I'm currently create a app.yaml file and run-microservice.sh file to run the service and collector. It seems not working. Do I need to create DockerFile or artifactory ?

[psx95]

I see, I will suggest to try with docker as well, although if I remember correctly, to deploy docker containers on app engine, you need to use App engine Flex instead of standard.

You can find the docs here. I did not get much time to look at it today, but I will try it on my end once as well.

[psx95]

I ran your example and looking at the logs, it seems like you will not be able to install the collector directly on the app engine environment. The logs suggest that it's a read only system and you are attempting to download files directly on it, so collector is not really running -

From this, I think containerizing your deployment through docker could work. Also, does your use-case specifically require you to use App Engine ?

[owensulei3841]

Hi Paramar, Thanks a lot for your help. Currently I’m just doing POC using an app engine. We may move the GKE in the future, currently we just want to make it workable using simple instances like app engines. I’m really new to docker, when you say containerizing my deployment through docker, what's the steps to do that? So I need to create an app.yaml file with flex mode, and DockerFile to install the docker container? Should I still need to download the binary files we downloaded in my local environment? I really appreciated it. Thanks a lot

Kevin

Pranav Sharma @.***>于2023年4月22日 周六13:40写道：

I ran your example and looking at the logs, it seems like you will not be able to install the collector directly on the app engine environment. The logs suggest that it's a read only system and you are attempting to download files directly on it, so collector is not really running - [image: image] https://user-images.githubusercontent.com/22191775/233805603-9ed9a406-df76-4b29-a49f-0a98e0cc1a52.png

From this, I think containerizing your deployment through docker could work. Also, does your use-case specifically require you to use App Engine ?

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1518774590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUCZTPFWWKHCIOMBRR3XCQ62HANCNFSM6AAAAAAXCAJAAU . You are receiving this because you authored the thread.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1518774590@ github.com>

[owensulei3841]

Hi Paramar, Now, I'm able to manually run the collector on the app engine after I set my app.yaml to "flex" mode. But I'm wondering if there is a way I can let the collector run automatically after the server starts. Can I ask if there is a example for that? Currently, I'm trying to modify my app.yaml and Dockerfile in the project to achieve that. But I got following error when run "gcloud app deploy --image-url gcr.io/alpine-eon-381904/web-site:v1"

ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2023-04-25T04:19:38.241Z3048.pl.2: exec /bin/sh: exec format error

I have create a Dockerfile like following, and I create a image and push that to my gcr

I also update my app.yaml to flex mode like following:

It seems I can run the image on local (but throw me exception after about permission issue), but when I try to use this image on gcp app engine, it fails like:

Can I ask do you know how to debug this? Or do you have a example I can follow to let my collector run automatically? Thank you

[psx95]

In order to run both Collector and your Spring boot project, you will need to use a custom runtime instead of java.
Right now, I don't think its actually running your Docker container.

You can refer to nginx custom runtime example for steps to setup.

You should also take a look at -

• Documentation to build custom runtime
• Documentation to create app running on custom runtime

Looking at your code, I would change the contents of app.yaml to simply -

runtime: custom
env: flex

And I noticed that in you Dockerfile your collector is not running in background. You have used &&, which I think will wait for the previous process to complete before starting a new one.

Not completely sure, but most likely you will need to change that line to -

CMD  ./otelcol-contrib --config /app/config.yaml & 
CMD   java -jar observability-poc-1.0.jar

'&' should put the process in background mode

[psx95]

Hi @owensulei3841, were you able to resolve this issue ?

[owensulei3841]

Yes，Pranav, I think right now I create a flex app.yaml and Dockerfile that downloads collector binary files in the app engine vim. I can see the data from the gcp, but I’m not sure it’s the best way to achieve. It takes 10-15minites to deploy my project to gcp using gcloud app deploy. I’m not sure if that’s right.

Pranav Sharma @.***>于2023年5月2日 周二08:13写道：

Hi @owensulei3841 https://github.com/owensulei3841, were you able to resolve this issue ?

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1531655640, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUD52TCYZPFWQZSTIXLXEEQCZANCNFSM6AAAAAAXCAJAAU . You are receiving this because you were mentioned.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1531655640@ github.com>

[psx95]

It could take more time to deploy if the base image used in the container is too big - other than that I am assuming it downloads the collector binary and installs it - which might be taking time. I am not sure of a better way as of now, but I'm glad you were able to export your telemetry data using GAE flex.

I will be closing this issue now. Feel free to re-open in case you still have any pending questions about telemetry exports.

[owensulei3841]

Thanks a lot Pranav. I’m really appreciated!

Pranav Sharma @.***>于2023年5月2日 周二09:53写道：

It could take more time to deploy if the base image used in the container is too big - other than that I am assuming it downloads the collector binary and installs it - which might be taking time. I am not sure of a better way as of now, but I'm glad you were able to export your telemetry data using GAE flex.

I will be closing this issue now. Feel free to re-open in case you still have any pending questions about telemetry exports.

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/232#issuecomment-1531817026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICSTUFRLJFAEQGQZBYOAOLXEE3YRANCNFSM6AAAAAAXCAJAAU . You are receiving this because you were mentioned.Message ID: <GoogleCloudPlatform/opentelemetry-operations-java/issues/232/1531817026@ github.com>