search

gnocchixyz / gnocchi

Timeseries database
Apache License 2.0
299 stars 85 forks source link

Rate computing can returns random results #244

Closed jd closed 7 years ago

jd commented 7 years ago

Travis hit random bugs like this:


==============================
Failed 2 tests - output below:
==============================

gabbi.suitemaker.test_gabbi_metric-derived_get_measurements_rate:mean.test_request
----------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 93, in wrapper'
    b'    func(self)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 141, in test_request'
    b'    self._run_test()'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 499, in _run_test'
    b'    self._assert_response()'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 186, in _assert_response'
    b'    handler(self)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/handlers/base.py", line 54, in __call__'
    b'    self.action(test, item, value=value)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/handlers/jsonhandler.py", line 109, in action'
    b'    (path, expected, match))'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual'
    b'    self.assertThat(observed, matcher, message)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat'
    b'    raise mismatch_error'
    b'testtools.matchers._impl.MismatchError: !=:'
    b"reference = [['2015-03-06T14:34:00+00:00', 60.0, 2.0],"
    b" ['2015-03-06T14:35:00+00:00', 60.0, 5.833333333333333],"
    b" ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]"
    b"actual    = [['2015-03-06T14:34:00+00:00', 60.0, 2.0],"
    b" ['2015-03-06T14:35:00+00:00', 60.0, 6.6],"
    b" ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]"
    b": Unable to match $ as [['2015-03-06T14:34:00+00:00', 60.0, 2.0], ['2015-03-06T14:35:00+00:00', 60.0, 5.833333333333333], ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]], got [['2015-03-06T14:34:00+00:00', 60.0, 2.0], ['2015-03-06T14:35:00+00:00', 60.0, 6.6], ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]"
    b''

gabbi.suitemaker.test_gabbi_metric-derived_get_measurements_rate:95pct.test_request
-----------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 93, in wrapper'
    b'    func(self)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 141, in test_request'
    b'    self._run_test()'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 499, in _run_test'
    b'    self._assert_response()'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/case.py", line 186, in _assert_response'
    b'    handler(self)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/handlers/base.py", line 54, in __call__'
    b'    self.action(test, item, value=value)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/gabbi/handlers/jsonhandler.py", line 109, in action'
    b'    (path, expected, match))'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual'
    b'    self.assertThat(observed, matcher, message)'
    b'  File "/home/tester/src/.tox/py35-postgresql/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat'
    b'    raise mismatch_error'
    b'testtools.matchers._impl.MismatchError: !=:'
    b"reference = [['2015-03-06T14:34:00+00:00', 60.0, 3.0],"
    b" ['2015-03-06T14:35:00+00:00', 60.0, 15.0],"
    b" ['2015-03-06T14:36:00+00:00', 60.0, 30.25]]"
    b"actual    = [['2015-03-06T14:34:00+00:00', 60.0, 3.0],"
    b" ['2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998],"
    b" ['2015-03-06T14:36:00+00:00', 60.0, 30.25]]"
    b": Unable to match $ as [['2015-03-06T14:34:00+00:00', 60.0, 3.0], ['2015-03-06T14:35:00+00:00', 60.0, 15.0], ['2015-03-06T14:36:00+00:00', 60.0, 30.25]], got [['2015-03-06T14:34:00+00:00', 60.0, 3.0], ['2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998], ['2015-03-06T14:36:00+00:00', 60.0, 30.25]]"
    b''
jd commented 7 years ago

Happened on Python 2.7 today: https://s3.amazonaws.com/archive.travis-ci.org/jobs/258214230/log.txt?X-Amz-Expires=30&X-Amz-Date=20170727T175053Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170727/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4a3660c80bf534c790b0d38b49410e5092605bedee7bd239205680d13251b4


==============================
Failed 1 tests - output below:
==============================

gabbi.suitemaker.test_gabbi_metric-derived_get_measurements_rate:95pct.test_request
-----------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
        func(self)
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
        self._run_test()
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/case.py", line 499, in _run_test
        self._assert_response()
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/case.py", line 186, in _assert_response
        handler(self)
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/handlers/base.py", line 54, in __call__
        self.action(test, item, value=value)
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/gabbi/handlers/jsonhandler.py", line 109, in action
        (path, expected, match))
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/tester/src/.tox/py27-mysql/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: !=:
    reference = [['2015-03-06T14:34:00+00:00', 60.0, 3.0],
     ['2015-03-06T14:35:00+00:00', 60.0, 15.0],
     ['2015-03-06T14:36:00+00:00', 60.0, 30.25]]
    actual    = [[u'2015-03-06T14:34:00+00:00', 60.0, 3.0],
     [u'2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998],
     [u'2015-03-06T14:36:00+00:00', 60.0, 30.25]]
    : Unable to match $ as [['2015-03-06T14:34:00+00:00', 60.0, 3.0], ['2015-03-06T14:35:00+00:00', 60.0, 15.0], ['2015-03-06T14:36:00+00:00', 60.0, 30.25]], got [[u'2015-03-06T14:34:00+00:00', 60.0, 3.0], [u'2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998], [u'2015-03-06T14:36:00+00:00', 60.0, 30.25]]```
jd commented 7 years ago

@sileht did we make any progress on this? I'm worried as it'll be a blocker for 4.1

jd commented 7 years ago

@sileht this is still happening https://s3.amazonaws.com/archive.travis-ci.org/jobs/271696433/log.txt?X-Amz-Expires=29&X-Amz-Date=20170904T143131Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170904/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=2f3481c637383e6510629d412dfe776097446a5a7c0a8183e038a34d0a00237d

sileht commented 7 years ago

I just added this tests https://gist.github.com/sileht/4386d96415f5076a37f83b02fc33dd68 and can't reproduce it with carbonara only it seems.

jd commented 7 years ago

Note another failure, pasting results here.


==============================
Failed 2 tests - output below:
==============================

gabbi.suitemaker.test_gabbi_metric-derived_get_measurements_rate:mean.test_request
----------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
        func(self)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
        self._run_test()
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 499, in _run_test
        self._assert_response()
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 186, in _assert_response
        handler(self)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/handlers/base.py", line 54, in __call__
        self.action(test, item, value=value)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/handlers/jsonhandler.py", line 109, in action
        (path, expected, match))
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: !=:
    reference = [['2015-03-06T14:34:00+00:00', 60.0, 2.0],
     ['2015-03-06T14:35:00+00:00', 60.0, 5.833333333333333],
     ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]
    actual    = [[u'2015-03-06T14:34:00+00:00', 60.0, 2.0],
     [u'2015-03-06T14:35:00+00:00', 60.0, 6.6],
     [u'2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]
    : Unable to match $ as [['2015-03-06T14:34:00+00:00', 60.0, 2.0], ['2015-03-06T14:35:00+00:00', 60.0, 5.833333333333333], ['2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]], got [[u'2015-03-06T14:34:00+00:00', 60.0, 2.0], [u'2015-03-06T14:35:00+00:00', 60.0, 6.6], [u'2015-03-06T14:36:00+00:00', 60.0, 9.166666666666666]]

gabbi.suitemaker.test_gabbi_metric-derived_get_measurements_rate:95pct.test_request
-----------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
        func(self)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
        self._run_test()
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 499, in _run_test
        self._assert_response()
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/case.py", line 186, in _assert_response
        handler(self)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/handlers/base.py", line 54, in __call__
        self.action(test, item, value=value)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/gabbi/handlers/jsonhandler.py", line 109, in action
        (path, expected, match))
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/tester/src/.tox/py27-postgresql/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: !=:
    reference = [['2015-03-06T14:34:00+00:00', 60.0, 3.0],
     ['2015-03-06T14:35:00+00:00', 60.0, 15.0],
     ['2015-03-06T14:36:00+00:00', 60.0, 30.25]]
    actual    = [[u'2015-03-06T14:34:00+00:00', 60.0, 3.0],
     [u'2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998],
     [u'2015-03-06T14:36:00+00:00', 60.0, 30.25]]
    : Unable to match $ as [['2015-03-06T14:34:00+00:00', 60.0, 3.0], ['2015-03-06T14:35:00+00:00', 60.0, 15.0], ['2015-03-06T14:36:00+00:00', 60.0, 30.25]], got [[u'2015-03-06T14:34:00+00:00', 60.0, 3.0], [u'2015-03-06T14:35:00+00:00', 60.0, 15.599999999999998], [u'2015-03-06T14:36:00+00:00', 60.0, 30.25]]

What intrigues me is that the values that are wrong are always wrong in the same way.