Failing tasks when monitoring all services

[mfranczak]

I used the Function (v0.0.23) deployment with all services and all metrics. Once run I started getting errors in logs. I would expect the lines to have Error severity for better filtering.

Please take a look:

insertId,labels.execution_id,logName,receiveTimestamp,resource.labels.function_name,resource.labels.project_id,resource.labels.region,resource.type,severity,textPayload,timestamp,trace
000000-03466100-899b-4226-a146-1a43c30aa2bc,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:16.359508 [dynatrace-gcp-extension] : Finished fetching data in 8.522547960281372,2021-02-22T12:59:16.359Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-03466100-899b-4226-a146-1a43c30aa2bc,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:15.691604 [dynatrace-gcp-extension] : Failed to finish task for [spanner.googleapis.com/instance/backup/used_bytes], reason is Exception {'error': {'code': 500, 'message': 'Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.', 'status': 'INTERNAL'}}",2021-02-22T12:59:15.691Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-f117c1db-c0d1-41d6-89bd-7871fad381b1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:15.083628 [dynatrace-gcp-extension] : Failed to finish task for [serviceruntime.googleapis.com/quota/limit], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:15.083Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-f117c1db-c0d1-41d6-89bd-7871fad381b1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:14.660401 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_processing_latencies], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/eventing/trigger/event_processing_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:14.660Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-7937c9c6-bd48-463b-9f72-1b4568ebc0d1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.691595 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_dispatch_latencies], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/eventing/trigger/event_dispatch_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.691Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-7937c9c6-bd48-463b-9f72-1b4568ebc0d1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.675049 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_count], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/eventing/trigger/event_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.675Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.045363 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/revision/request_latencies], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/serving/revision/request_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.045Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.043511 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/revision/request_count], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/serving/revision/request_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.043Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.041706 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/activator/request_latencies], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/serving/activator/request_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.041Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.040909 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/activator/request_count], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/serving/activator/request_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.040Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.038194 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/broker/event_count], reason is Exception {'error': {'code': 403, 'message': 'User dynatrace-gcp-service@dynatrace-gcp-extension.iam.gserviceaccount.com does not have permission to see metric knative.dev/internal/eventing/broker/event_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.038Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-40cb4b5e-f754-46b6-b50b-0ed1b8025069,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:12.656580 [dynatrace-gcp-extension] : Failed to finish task for [kubernetes.io/container/memory/page_fault_count], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.perSeriesAligner had an invalid value of ""ALIGN_SUM"": The aligner cannot be applied to metrics with kind CUMULATIVE and value type INT64.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:12.656Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-ef4753e0-65d0-4b9e-a074-57cc4ab8d3e1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.754083 [dynatrace-gcp-extension] : Failed to finish task for [firebasedatabase.googleapis.com/storage/disabled_for_overages], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.754Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-ef4753e0-65d0-4b9e-a074-57cc4ab8d3e1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.700913 [dynatrace-gcp-extension] : Failed to finish task for [firebasedatabase.googleapis.com/storage/total_bytes], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.701Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-cd235d69-db4a-4d64-a882-c54e761f93e4,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.655435 [dynatrace-gcp-extension] : Failed to finish task for [serviceruntime.googleapis.com/quota/limit], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.655Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.837048 [dynatrace-gcp-extension] : Starting processing...,2021-02-22T12:59:07.837Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.836718  : Access to following projects: dynatrace-gcp-extension,2021-02-22T12:59:07.836Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.485475  : Successfully obtained access token,2021-02-22T12:59:07.485Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000001-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.462862  : Trying to use default service account,2021-02-22T12:59:07.462Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-cc188f3e-7061-4623-b5ad-ae7784ddf6e6,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:07.462572  : Selected feature sets: api/default,apigee.googleapis.com/Environment/default,apigee.googleapis.com/Proxy/default,apigee.googleapis.com/ProxyV2/default,assistant_action_project/default,autoscaler/default,bigquery_biengine_model/default,bigquery_dataset/default,bigquery_project/default,bigtable_cluster/default,bigtable_table/default,cloud_composer_environment/default,cloud_dataproc_cluster/default,cloud_dlp_project/default,Cloud Function/default,cloud_run_revision/default,cloud_tasks_queue/default,cloudiot_device_registry/default,cloudml_job/default,cloudml_model_version/default,cloudsql_database/default,cloudtrace.googleapis.com/CloudtraceProject/default,cloudvolumesgcp-api.netapp.com/NetAppCloudVolumeSO/default,consumed_api/default,consumer_quota/default,dataflow_job/default,datastore_request/default,dns_query/default,filestore_instance/default,firebase_domain/default,firebase_namespace/default,firestore_instance/default,gae_app/default,gae_app_uptime_check/default,gae_instance/default,gce_instance/default,gce_instance/agent,gce_instance/appenginee,gce_instance/firewallinsights,gce_instance/istio,gce_instance/uptime_check,gce_instance_vm_flow/,gce_router/default,gce_zone_network_health/default,gcs_bucket/default,gke_container/default,https_lb_rule/default,iam_service_account/default,instance_group/default,interconnect/default,interconnect_attachment/default,internal_http_lb_rule/default,internal_tcp_lb_rule/default,internal_udp_lb_rule/default,istio_canonical_service/default,k8s_cluster/default,k8s_container/default,k8s_container/agent,k8s_container/apigee,k8s_container/istio,k8s_container/nginx,k8s_node/default,k8s_pod/default,knative_broker/default,knative_revision/default,knative_trigger/default,logging_sink/default,microsoft_ad_domain/default,nat_gateway/default,netapp_cloud_volume/default,network_security_policy/default,producer_quota/default,pubsub_snapshot/default,pubsub_subscription/default,pubsub_topic/default,pubsublite_subscription_partition/default,pubsublite_topic_partition/default,recaptchaenterprise.googleapis.com/Key/default,redis_instance/default,spanner_instance/default,tcp_lb_rule/default,tcp_ssl_proxy_rule/default,tpu_worker/default,transfer_service_agent/default,udp_lb_rule/default,uptime_url/default,vpc_access_connector/default,vpn_gatewayv/",2021-02-22T12:59:07.462Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38

[pgutkowski]

@mfranczak Thank you for the report, I've investigated the issues with metrics, and we have 3 separate problems here:

• Errors with Field aggregation.alignmentPeriod had an invalid value of ""0"" message - reason is minor bug in calculated sample period timedelta. will be fixed with #61
• kubernetes.io/container/memory/page_fault_count metric has wrong metricType, Will be fixed with #61 as well.
• metrics from knative.dev/internal namespace are in alpha/beta stage and probably are not available in your monitored project. No fix required there.