open-telemetry / opentelemetry-python-contrib

OpenTelemetry instrumentation for Python modules
https://opentelemetry.io
Apache License 2.0
705 stars 589 forks source link

Flaky flask tests with pypy in CI #2883

Closed xrmx closed 2 weeks ago

xrmx commented 2 weeks ago

Describe your environment

OS: (e.g, Ubuntu) Python version: pypy-3.8 Package version: 6fcfbd67b85e0139147b3273da97c30c4a3970a8

What happened?

_______________ TestProgrammatic.test_flask_metrics_new_semconv ________________

self = <tests.test_programmatic.TestProgrammatic testMethod=test_flask_metrics_new_semconv>

    def test_flask_metrics_new_semconv(self):
        start = default_timer()
        self.client.get("/hello/123")
        self.client.get("/hello/321")
        self.client.get("/hello/756")
        duration_s = max(default_timer() - start, 0)
        metrics_list = self.memory_metrics_reader.get_metrics_data()
        number_data_point_seen = False
        histogram_data_point_seen = False
        self.assertTrue(len(metrics_list.resource_metrics) != 0)
        for resource_metric in metrics_list.resource_metrics:
            self.assertTrue(len(resource_metric.scope_metrics) != 0)
            for scope_metric in resource_metric.scope_metrics:
                self.assertTrue(len(scope_metric.metrics) != 0)
                for metric in scope_metric.metrics:
                    self.assertIn(metric.name, _expected_metric_names_new)
                    data_points = list(metric.data.data_points)
                    self.assertEqual(len(data_points), 1)
                    for point in data_points:
                        if isinstance(point, HistogramDataPoint):
                            self.assertEqual(point.count, 3)
>                           self.assertAlmostEqual(
                                duration_s, point.sum, places=2
                            )
E                           AssertionError: 0.018902038000021548 != 0.013057114999980968 within 2 places (0.0058449230000405805 difference)

Steps to Reproduce

tox -e pypy3-test-instrumentation-flask-1 -- -ra

Expected Result

pass

Actual Result

fail

Additional context

No response

Would you like to implement a fix?

No

xrmx commented 2 weeks ago

Another try:

>>> round(0.0012783256999966852, 2)
0.0
>>> round(0.007484300999976767, 2)
0.01

The fix would be to use places=1 or delta I guess