search

googleapis / google-cloud-ruby

Google Cloud Client Library for Ruby
https://googleapis.github.io/google-cloud-ruby/
Apache License 2.0
1.35k stars 543 forks source link

[Test failure] Monitoring samples test flake #19988

Open dazuma opened 1 year ago

dazuma commented 1 year ago 
1) Error:
metrics::monitoring::delete_metric_descriptor#test_0001_deletes a metric descriptor:
Google::Cloud::NotFoundError: 5:The metric '[custom.googleapis.com/my_metric_b18tl9g5kn](https://www.google.com/url?q=http://custom.googleapis.com/my_metric_b18tl9g5kn&sa=D)' does not exist.. debug_error_string:{UNKNOWN:Error received from peer ipv4:74.125.142.95:443 {grpc_message:"The metric \'[custom.googleapis.com/my_metric_b18tl9g5kn\](https://www.google.com/url?q=http://custom.googleapis.com/my_metric_b18tl9g5kn%5C&sa=D)' does not exist.", grpc_status:5, created_time:"2023-01-01T10:39:01.991609692+00:00"}}
    /[h/.gem/gems/google-cloud-monitoring-v3-0.9.0/lib/google/cloud/monitoring/v3/metric_service/client.rb:764](https://cs.corp.google.com/piper///depot/google3/h/.gem/gems/google-cloud-monitoring-v3-0.9.0/lib/google/cloud/monitoring/v3/metric_service/client.rb?l=764):in `rescue in delete_metric_descriptor'
    /[h/.gem/gems/google-cloud-monitoring-v3-0.9.0/lib/google/cloud/monitoring/v3/metric_service/client.rb:726](https://cs.corp.google.com/piper///depot/google3/h/.gem/gems/google-cloud-monitoring-v3-0.9.0/lib/google/cloud/monitoring/v3/metric_service/client.rb?l=726):in `delete_metric_descriptor'
    /[workspace/google-cloud-monitoring/samples/metrics.rb:54](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/metrics.rb?l=54):in `delete_metric_descriptor'
    /[workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb:47](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb?l=47):in `block (5 levels) in <top (required)>'
    /[workspace/google-cloud-monitoring/samples/acceptance/helper.rb:37](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/helper.rb?l=37):in `block in retry_block'
    /[workspace/google-cloud-monitoring/samples/acceptance/helper.rb:35](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/helper.rb?l=35):in `loop'
    /[workspace/google-cloud-monitoring/samples/acceptance/helper.rb:35](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/helper.rb?l=35):in `retry_block'
    /[workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb:46](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb?l=46):in `block (4 levels) in <top (required)>'

This might be a case of one test run's cleanup stomping on another test run.

dazuma commented 1 year ago

Saw a similar flake:

  1) Error:
metrics::monitoring::create_metric_descriptor#test_0001_creates a metric descriptor:
Google::Cloud::NotFoundError: 5:The metric 'custom.googleapis.com/my_metric_2w81a5elrj' does not exist.. debug_error_string:{UNKNOWN:Error received from peer ipv4:74.125.142.95:443 {created_time:"2023-01-12T07:56:15.205259607+00:00", grpc_status:5, grpc_message:"The metric \'custom.googleapis.com/my_metric_2w81a5elrj\' does not exist."}}
    /workspace/google-cloud-monitoring-v3/lib/google/cloud/monitoring/v3/metric_service/client.rb:764:in `rescue in delete_metric_descriptor'
    /workspace/google-cloud-monitoring-v3/lib/google/cloud/monitoring/v3/metric_service/client.rb:726:in `delete_metric_descriptor'
    /workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb:24:in `block (3 levels) in <top (required)>'
dazuma commented 1 year ago

Haven't seen the original report in a while, but the more recent one is appearing every few days:

1) Error:
metrics::monitoring::create_metric_descriptor#test_0001_creates a metric descriptor:
Google::Cloud::NotFoundError: 5:The metric '[custom.googleapis.com/my_metric_k3lresxhp](https://www.google.com/url?q=http://custom.googleapis.com/my_metric_k3lresxhp&sa=D)' does not exist.. debug_error_string:{UNKNOWN:Error received from peer ipv4:74.125.199.95:443 {grpc_message:"The metric \'[custom.googleapis.com/my_metric_k3lresxhp\](https://www.google.com/url?q=http://custom.googleapis.com/my_metric_k3lresxhp%5C&sa=D)' does not exist.", grpc_status:5, created_time:"2023-05-12T08:58:27.191012469+00:00"}}
    /[h/.gem/gems/google-cloud-monitoring-v3-0.11.0/lib/google/cloud/monitoring/v3/metric_service/client.rb:760](https://cs.corp.google.com/piper///depot/google3/h/.gem/gems/google-cloud-monitoring-v3-0.11.0/lib/google/cloud/monitoring/v3/metric_service/client.rb?l=760):in `rescue in delete_metric_descriptor'
    /[h/.gem/gems/google-cloud-monitoring-v3-0.11.0/lib/google/cloud/monitoring/v3/metric_service/client.rb:722](https://cs.corp.google.com/piper///depot/google3/h/.gem/gems/google-cloud-monitoring-v3-0.11.0/lib/google/cloud/monitoring/v3/metric_service/client.rb?l=722):in `delete_metric_descriptor'
    /[workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb:24](https://cs.corp.google.com/piper///depot/google3/workspace/google-cloud-monitoring/samples/acceptance/metrics_test.rb?l=24):in `block (3 levels) in <top (required)>'

I suspect the reported error, which is in cleanup code (an after block) is a symptom of the fact that the actual sample (which is create_metric_descriptor in this case is failing for other reasons. I think we should suppress the cleanup error so the actual error gets reported correctly.