search

dynatrace-oss / dynatrace-gcp-monitor

Dynatrace integration for Google Cloud Platform monitoring
https://www.dynatrace.com/support/help/technology-support/cloud-platforms/google-cloud-platform/
Apache License 2.0
34 stars 20 forks source link

Failed to push self monitoring time series #25

Closed hom-bahrani closed 3 years ago

hom-bahrani commented 3 years ago

I followed the Kubernetes quick start in the readme on a simple cluster, when I check the container status I get a 403 permissions denied error. Any idea what this might be? When I follow the bash instructions and deploy a cloud function it works fine and I'm seeing metrics in Dynatrace, but I have some constraints and have to go via the kubernetes route, I used the same Dynatrace API Token in both cases. Many thanks in advance

kubectl -n dynatrace logs -l app=dynatrace-gcp-function

2020-11-02 17:06:34.795092 [c5410aad] : Finished uploading metric ingest lines to Dynatrace in 1.1920928955078125e-06 s
2020-11-02 17:06:34.795097 [c5410aad] : Pushing self monitoring time series to GCP Monitor...
2020-11-02 17:06:34.889516 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-02 17:06:34.988816 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-02 17:06:35.086464 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-02 17:06:35.186966 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-02 17:06:35.285764 [c5410aad] : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denie
d (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-02 17:06:35.286386 [c5410aad] : Execution took 2.5885870456695557
pawelsiwek commented 3 years ago

HI @hom-bahrani , Can you double check if You have granted roles/monitoring.editor to the Service Account.

Some background: dynatrace-gcp-function tries to register self-monitoring metrics in GCP to help with troubleshooting if there is no data in Dynatrace. 403's You are getting are related to this feature: function does not have permission to create metrics descriptors. This issue will not stop pushing metrics to Dynatrace, only self-monitoring in Google Operations will not work..

hom-bahrani commented 3 years ago

Hi @pawelsiwek many thanks for the quick reply, I gave the dynatrace-gcp-function service account project editor permissions as quick test but I'm still not seeing any self-monitoring in Google Operations or getting any metrics pushed to Dynatrace. Also I double checked the Cloud Function installation again, as before its pushing all the metrics using my Dynatrace tenant URI and API token. I also inspected the container logs in my dynatrace-gcp-function pod running on my cluster and I don't see the nice output that I get at the end of the cloud function installation confirming that all the dashboards have been created for me, is that to be expected?

- Create ["Google Cloud APIs"] dashboard from file [dashboards/api.json]
{"id":"5010d65b-559c-4b7b-baab-367b2a983aa9","name":"Google Cloud APIs"}
- Create ["Google Cloud Function"] dashboard from file [dashboards/cloud_function.json]
{"id":"06e8d1f7-e8fa-4d67-80cc-0615d3f35708","name":"Google Cloud Function"}
- Create ["Google Cloud SQL"] dashboard from file [dashboards/cloudsql_database.json]
{"id":"3487cfc7-8ff0-4b0a-b4fd-dd2da706d400","name":"Google Cloud SQL"}
- Create ["Google Cloud Datastore"] dashboard from file [dashboards/datastore.json]
{"id":"1ae5e77b-399b-4318-a848-cca3b6ea0534","name":"Google Cloud Datastore"}
- Create ["Google Cloud Filestore"] dashboard from file [dashboards/filestore_instance.json]
{"id":"4c0ac430-8bd0-45c9-a70f-c94c9e4392f3","name":"Google Cloud Filestore"}
- Create ["Google Cloud Storage"] dashboard from file [dashboards/gcs_bucket.json]
{"id":"42c606b9-9ace-4083-8b7f-b58892081303","name":"Google Cloud Storage"}
- Create ["Google Cloud HTTPs Load Balancing"] dashboard from file [dashboards/lb_https.json]
{"id":"19bbd395-fcee-4148-91d7-44afdff51710","name":"Google Cloud HTTPs Load Balancing"}
- Create ["Google Cloud TCP Load Balancing"] dashboard from file [dashboards/lb_tcp.json]
{"id":"4ff9840b-fd08-43af-94d3-2b37218b68f4","name":"Google Cloud TCP Load Balancing"}
- Create ["Google Cloud Pub/Sub"] dashboard from file [dashboards/pubsub.json]
{"id":"44bddef7-1cf8-41df-b21c-66f8bf41c282","name":"Google Cloud Pub/Sub"}
pawelsiwek commented 3 years ago

Hi, indeed K8S approach is not (currently) importing dashboards to Dynatrace. We will add initContainer to k8s Deployment template that will do the trick. I observed that sometimes pods logs appears after some time in GKE. Can you verify if container logs are there now?

hom-bahrani commented 3 years ago

thanks @pawelsiwek yes you're right the pod logs appear after a few minutes

pawelsiwek commented 3 years ago

@hom-bahrani are the metrics flying to Dynatrace or still U have issue?

hom-bahrani commented 3 years ago

Hi @pawelsiwek thanks for the update. Unfortunately not! Strangely the error from before for self monitoring metrics has not gone either, even with the roles/monitoring.editor role 

2020-11-03 17:08:36.606911 [834eb7af] : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
pgutkowski commented 3 years ago

@hom-bahrani could you please share with us more of the logs? Please run kubectl -n dynatrace logs --tail=1000 -l app=dynatrace-gcp-function so we will be able to see at least one full run of the worker.

hom-bahrani commented 3 years ago

@pgutkowski here are the steps to replicate, logs below

gcloud config set compute/zone us-central1-c

gcloud services enable container.googleapis.com
# Operation "operations/acf.4ba29857-d879-4138-8a48-cb8cdea16bfa" finished successfully.

gcloud container clusters create "demo-cluster" \
  --num-nodes "1" \
  --machine-type "n1-standard-1" \
  --cluster-version "1.16.13-gke.401" \
  --zone "us-central1-c" \
  --no-enable-basic-auth \
  --image-type "COS" \
  --disk-type "pd-standard" \
  --disk-size "100" \
  --enable-stackdriver-kubernetes \
  --enable-ip-alias
# NAME          LOCATION       MASTER_VERSION   MASTER_IP       MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
# demo-cluster  us-central1-c  1.16.13-gke.401  35.224.179.208  n1-standard-1  1.16.13-gke.401  1          RUNNING

gcloud container clusters get-credentials demo-cluster
# kubeconfig entry generated for demo-cluster.

kubectl create namespace dynatrace
# namespace/dynatrace created

kubectl -n dynatrace create secret generic dynatrace-gcp-function-secret --from-literal="access-key={DYNATRACE_API_TOKEN}" --from-literal="url={DYNATRACE_URL}"
# secret/dynatrace-gcp-function-secret created

gcloud iam service-accounts create dynatrace-gcp-function-sa --display-name "dynatrace-gcp-function-sa"
# Created service account [dynatrace-gcp-function-sa]

gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/monitoring.editor
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/compute.viewer
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/cloudsql.viewer
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/cloudfunctions.viewer
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/file.viewer
gcloud projects add-iam-policy-binding playground-s-11-5f1276bb --member="serviceAccount:dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com" --role=roles/pubsub.viewer

gcloud services enable cloudapis.googleapis.com monitoring.googleapis.com
# Operation "operations/acf.80d1fa7e-2f31-4ffa-956c-c46c7f7ce759" finished successfully.

wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-function/master/k8s/dynatrace-gcp-function.yaml

kubectl apply -f dynatrace-gcp-function.yaml
# serviceaccount/dynatrace-gcp-function-sa created
# configmap/dynatrace-gcp-function-config created
# deployment.apps/dynatrace-gcp-function created

kubectl annotate serviceaccount --namespace dynatrace dynatrace-gcp-function-sa iam.gke.io/gcp-service-account=dynatrace-gcp-function-sa@playground-s-11-5f1276bb.iam.gserviceaccount.com
# serviceaccount/dynatrace-gcp-function-sa annotated

kubectl -n dynatrace logs --tail=1000 -l app=dynatrace-gcp-function

Logs

                    ,,,,,..
                  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,.
               ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
            .,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,     ,,
          ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,    .,,,,
       ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,     ,,,,,,,.
    .,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,.    ,,,,,,,,,,,
  ,,,,,,,,,,,,,,,,,......  ......,,,,,,,    .,,,,,,,,,,,,,
,,                                        ,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,                        .,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,                        .,,,,,,,,,,,,,,,,.
,,,,,,,,,,,,,,,,,       Dynatrace        .,,,,,,,,,,,,,,,,.
,,,,,,,,,,,,,,,,, dynatrace-gcp-function .,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,                        .,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,                        ,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,                        ,,,,,,,,,,,,,,,,,,
.,,,,,,,,,,,,,,,                         ,,,,,,,,,,,,,,,,,
.,,,,,,,,,,,,,    .,,,,,,,,,,,,,,,,,,.   ,,,,,,,,,,,,,,,
 ,,,,,,,,,,     ,,,,,,,,,,,,,,,,,,,,,,  .,,,,,,,,,,,,.
 ,,,,,,,     ,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,,,,,
 ,,,,,    .,,,,,,,,,,,,,,,,,,,,,,,,,,.  ,,,,,,,,
  ,     ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,
     ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,

2020-11-05 10:51:51.343561 : Dynatrace function for Google Cloud Platform monitoring

Running with configured services: api,cloud_function,cloudsql_database,datastore_request,filestore_instance,gcs_bucket,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition
2020-11-05 10:51:51.343596 : Setting up... 

2020-11-05 10:51:51.344057 [4b3d1684] : Starting execution
2020-11-05 10:51:53.365444 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:51:53.366302 : Trying to use default service account
2020-11-05 10:51:53.466241 : Successfully obtained access token
2020-11-05 10:51:53.509698 : Access to following projects: 
2020-11-05 10:51:53.509756 : Fetched GCP data in 2.1696090698242188e-05 s
2020-11-05 10:51:53.509766 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:51:53.509771 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:51:53.610875 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:51:53.710088 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:51:53.809946 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:51:53.909637 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:51:54.010682 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:51:54.012354 [4b3d1684] : Execution took 2.6682727336883545

2020-11-05 10:52:51.384452 [6307e2aa] : Starting execution
2020-11-05 10:52:53.347520 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:52:53.348212 : Trying to use default service account
2020-11-05 10:52:53.378355 : Successfully obtained access token
2020-11-05 10:52:53.409606 : Access to following projects: 
2020-11-05 10:52:53.409653 : Fetched GCP data in 1.430511474609375e-05 s
2020-11-05 10:52:53.409663 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:52:53.409668 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:52:53.511596 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:52:53.611293 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:52:53.710174 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:52:53.809431 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:52:53.911759 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:52:53.912359 [6307e2aa] : Execution took 2.527869701385498

2020-11-05 10:53:51.427242 [49fb8e8d] : Starting execution
2020-11-05 10:53:53.428786 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:53:53.429565 : Trying to use default service account
2020-11-05 10:53:53.457597 : Successfully obtained access token
2020-11-05 10:53:53.486392 : Access to following projects: 
2020-11-05 10:53:53.486441 : Fetched GCP data in 1.2874603271484375e-05 s
2020-11-05 10:53:53.486451 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:53:53.486456 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:53:53.610230 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:53:53.712024 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:53:53.811439 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:53:53.910601 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:53:54.012138 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:53:54.012749 [49fb8e8d] : Execution took 2.5854671001434326

2020-11-05 10:54:51.483009 [e216dd12] : Starting execution
2020-11-05 10:54:53.460601 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:54:53.461252 : Trying to use default service account
2020-11-05 10:54:53.488429 : Successfully obtained access token
2020-11-05 10:54:53.515738 : Access to following projects: 
2020-11-05 10:54:53.515785 : Fetched GCP data in 1.1920928955078125e-05 s
2020-11-05 10:54:53.515796 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:54:53.515800 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:54:53.609756 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:54:53.712393 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:54:53.812488 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:54:53.910173 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:54:54.010861 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:54:54.011413 [e216dd12] : Execution took 2.5283656120300293

2020-11-05 10:55:51.540663 [fadc1eee] : Starting execution
2020-11-05 10:55:53.542299 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:55:53.543034 : Trying to use default service account
2020-11-05 10:55:53.565867 : Successfully obtained access token
2020-11-05 10:55:53.596873 : Access to following projects: 
2020-11-05 10:55:53.596949 : Fetched GCP data in 1.6689300537109375e-05 s
2020-11-05 10:55:53.596960 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:55:53.596965 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:55:53.710461 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:55:53.810138 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:55:53.911425 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:55:54.011066 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:55:54.110437 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:55:54.111077 [fadc1eee] : Execution took 2.5695698261260986

2020-11-05 10:56:51.596257 [928891e2] : Starting execution
2020-11-05 10:56:53.582624 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:56:53.584283 : Trying to use default service account
2020-11-05 10:56:53.623335 : Successfully obtained access token
2020-11-05 10:56:53.665620 : Access to following projects: 
2020-11-05 10:56:53.665667 : Fetched GCP data in 1.430511474609375e-05 s
2020-11-05 10:56:53.665678 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:56:53.665683 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:56:53.814676 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:56:53.910986 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:56:54.009634 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:56:54.109626 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:56:54.208556 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:56:54.209122 [928891e2] : Execution took 2.612828493118286

2020-11-05 10:57:51.648153 [7b2516b1] : Starting execution
2020-11-05 10:57:53.636201 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:57:53.637040 : Trying to use default service account
2020-11-05 10:57:53.684843 : Successfully obtained access token
2020-11-05 10:57:53.729284 : Access to following projects: 
2020-11-05 10:57:53.729336 : Fetched GCP data in 1.52587890625e-05 s
2020-11-05 10:57:53.729347 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:57:53.729352 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:57:53.811521 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:57:53.907834 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:57:54.008805 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:57:54.108301 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:57:54.209422 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:57:54.210007 [7b2516b1] : Execution took 2.5618138313293457

2020-11-05 10:58:51.694859 [a8dac70d] : Starting execution
2020-11-05 10:58:53.650246 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:58:53.650997 : Trying to use default service account
2020-11-05 10:58:53.686948 : Successfully obtained access token
2020-11-05 10:58:53.729447 : Access to following projects: 
2020-11-05 10:58:53.729497 : Fetched GCP data in 1.430511474609375e-05 s
2020-11-05 10:58:53.729507 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 10:58:53.729511 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:58:53.809539 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:58:53.909867 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:58:54.008558 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:58:54.106655 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:58:54.208519 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:58:54.209149 [a8dac70d] : Execution took 2.5142529010772705

2020-11-05 10:59:51.739974 [7d10f986] : Starting execution
2020-11-05 10:59:53.733112 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 10:59:53.733842 : Trying to use default service account
2020-11-05 10:59:53.769470 : Successfully obtained access token
2020-11-05 10:59:53.812272 : Access to following projects: 
2020-11-05 10:59:53.812319 : Fetched GCP data in 1.2636184692382812e-05 s
2020-11-05 10:59:53.812330 : Finished uploading metric ingest lines to Dynatrace in 9.5367431640625e-07 s
2020-11-05 10:59:53.812341 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 10:59:53.910398 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 10:59:54.009044 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 10:59:54.106608 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 10:59:54.207939 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 10:59:54.310609 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 10:59:54.311160 [7d10f986] : Execution took 2.5711495876312256

2020-11-05 11:00:51.799187 [346eb534] : Starting execution
2020-11-05 11:00:53.766890 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:00:53.768238 : Trying to use default service account
2020-11-05 11:00:53.821448 : Successfully obtained access token
2020-11-05 11:00:53.851242 : Access to following projects: 
2020-11-05 11:00:53.851292 : Fetched GCP data in 1.4781951904296875e-05 s
2020-11-05 11:00:53.851303 : Finished uploading metric ingest lines to Dynatrace in 4.76837158203125e-07 s
2020-11-05 11:00:53.851308 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 11:00:54.006780 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 11:00:54.104803 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 11:00:54.204243 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 11:00:54.307002 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 11:00:54.407122 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 11:00:54.407632 [346eb534] : Execution took 2.6084084510803223

2020-11-05 11:01:51.846118 [8b033f09] : Starting execution
2020-11-05 11:01:53.867604 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:01:53.868370 : Trying to use default service account
2020-11-05 11:01:53.961215 : Successfully obtained access token
2020-11-05 11:01:53.996120 : Access to following projects: 
2020-11-05 11:01:53.996173 : Fetched GCP data in 1.430511474609375e-05 s
2020-11-05 11:01:53.996185 : Finished uploading metric ingest lines to Dynatrace in 9.5367431640625e-07 s
2020-11-05 11:01:53.996190 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 11:01:54.105673 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 11:01:54.206583 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 11:01:54.305989 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 11:01:54.405251 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 11:01:54.506197 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 11:01:54.506753 [8b033f09] : Execution took 2.660597562789917

2020-11-05 11:02:51.904054 [5fb854d3] : Starting execution
2020-11-05 11:02:53.872840 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:02:53.873651 : Trying to use default service account
2020-11-05 11:02:53.908137 : Successfully obtained access token
2020-11-05 11:02:53.977059 : Access to following projects: 
2020-11-05 11:02:53.977109 : Fetched GCP data in 1.4543533325195312e-05 s
2020-11-05 11:02:53.977119 : Finished uploading metric ingest lines to Dynatrace in 4.76837158203125e-07 s
2020-11-05 11:02:53.977124 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 11:02:54.104620 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 11:02:54.207008 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 11:02:54.313285 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 11:02:54.408603 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 11:02:54.506519 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 11:02:54.507151 [5fb854d3] : Execution took 2.6030521392822266

2020-11-05 11:03:51.961386 [95489ede] : Starting execution
2020-11-05 11:03:53.947687 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:03:53.948378 : Trying to use default service account
2020-11-05 11:03:54.012341 : Successfully obtained access token
2020-11-05 11:03:54.054952 : Access to following projects: 
2020-11-05 11:03:54.055008 : Fetched GCP data in 1.5735626220703125e-05 s
2020-11-05 11:03:54.055021 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 11:03:54.055025 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 11:03:54.208709 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 11:03:54.308795 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 11:03:54.408472 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 11:03:54.508986 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 11:03:54.609789 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 11:03:54.610340 [95489ede] : Execution took 2.648916006088257

2020-11-05 11:04:52.019574 [c85581dc] : Starting execution
2020-11-05 11:04:54.049185 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:04:54.049992 : Trying to use default service account
2020-11-05 11:04:54.122497 : Successfully obtained access token
2020-11-05 11:04:54.159144 : Access to following projects: 
2020-11-05 11:04:54.159194 : Fetched GCP data in 1.4781951904296875e-05 s
2020-11-05 11:04:54.159203 : Finished uploading metric ingest lines to Dynatrace in 7.152557373046875e-07 s
2020-11-05 11:04:54.159208 : Pushing self monitoring time series to GCP Monitor...
2020-11-05 11:04:54.307043 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-05 11:04:54.409465 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-05 11:04:54.508119 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-05 11:04:54.600719 : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-05 11:04:54.707411 : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-05 11:04:54.707980 [c85581dc] : Execution took 2.688345432281494

2020-11-05 11:05:52.069248 [d55f7e0f] : Starting execution
2020-11-05 11:05:54.052835 : Selected services: cloud_function,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,cloudsql_database,datastore_request,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,api,gcs_bucket
2020-11-05 11:05:54.053524 : Trying to use default service account
2020-11-05 11:05:54.099219 : Successfully obtained access token
pgutkowski commented 3 years ago

@hom-bahrani thank you for the steps to replicate and logs. Your cluster setup misses argument to enable the workload identity, see https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_cluster. I'd recommend to remove the cluster and create a fresh one.

We'll update the README to communicate clearly that Workload Identity is required for k8s deployment.

hom-bahrani commented 3 years ago

@pgutkowski perfect, thanks so much