zopefoundation / zope.sendmail

Zope sendmail support.
Other
3 stars 6 forks source link

Flaky test: TestSMTPMailerWithNoEHLO.test_send_multiple_threads #15

Closed mgedmin closed 3 years ago

mgedmin commented 7 years ago

Examples of failures:

log ``` $ coverage run -m zope.testrunner --test-path=src Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 914, in _bootstrap_inner self.run() File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 147, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 1 Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Traceback (most recent call last): File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 59, in testPartExecutor yield File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 601, in run testMethod() File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1019, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual self.fail(msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 666, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)] First differing element 1: True AssertionError('2 != 1',) - [True, True] + [True, AssertionError('2 != 1',)] Ran 77 tests with 1 failures, 0 errors and 3 skipped in 2.738 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. ```
log ``` Running test-ztk-zope.sendmail test-ztk-zope.sendmail failed with: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/2.7.13/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/opt/python/2.7.13/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 147, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 1 Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Traceback (most recent call last): File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 742, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 724, in assertSequenceEqual self.fail(msg) File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 410, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1'... First differing element 1: True AssertionError('2 != 1',) - [True, True] + [True, AssertionError('2 != 1',)] Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.107 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Running test-ztk-zope.i18n ... ```
log ``` Running test-ztk-zope.sendmail test-ztk-zope.sendmail failed with: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/threading.py", line 797, in __bootstrap_inner self.run() File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/threading.py", line 750, in run self.__target(*self.__args, **self.__kwargs) File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 147, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 0 Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Traceback (most recent call last): File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 329, in run testMethod() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 742, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 724, in assertSequenceEqual self.fail(msg) File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 410, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 0'... First differing element 1: True AssertionError('2 != 0',) - [True, True] + [True, AssertionError('2 != 0',)] Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.224 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Running test-ztk-zope.session ... ```
log ``` Running test-ztk-zope.sendmail test-ztk-zope.sendmail failed with: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Traceback (most recent call last): File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 59, in testPartExecutor yield File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 601, in run testMethod() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1019, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual self.fail(msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 666, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)] First differing element 1: True AssertionError('2 != 1',) - [True, True] + [True, AssertionError('2 != 1',)] Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.263 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 914, in _bootstrap_inner self.run() File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 147, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 1 in 0.000 seconds. Running test-ztk-zope.session ... ```
icemac commented 4 years ago

This problem still seems to exist:

log ``` Running test-ztk-zope.sendmail test-ztk-zope.sendmail failed with: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailer) Traceback (most recent call last): File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 59, in testPartExecutor yield File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 601, in run testMethod() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 171, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1019, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual self.fail(msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 666, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)] First differing element 1: True AssertionError('2 != 1',) - [True, True] + [True, AssertionError('2 != 1',)] Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Traceback (most recent call last): File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 59, in testPartExecutor yield File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 601, in run testMethod() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 171, in test_send_multiple_threads self.assertEqual([True for _ in threads], results) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1019, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual self.fail(msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 666, in fail raise self.failureException(msg) AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)] First differing element 1: True AssertionError('2 != 1',) - [True, True] + [True, AssertionError('2 != 1',)] Ran 78 tests with 2 failures, 0 errors and 3 skipped in 0.156 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests Exception in thread Thread-1: Traceback (most recent call last): File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 155, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 146, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 1 Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 155, in run self.test_send_multiple_same_mailer() File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 146, in test_send_multiple_same_mailer self.assertEqual(2, len(smtps)) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual raise self.failureException(msg) AssertionError: 2 != 1 in 0.000 seconds. Running test-ztk-zope.session ... ```
mgedmin commented 3 years ago

Here's the test:

https://github.com/zopefoundation/zope.sendmail/blob/85e2af8ed2aaf7c31fc0c7a3e7482e99eadedd4e/src/zope/sendmail/tests/test_mailer.py#L148-L171

It calls self.test_send_multiple_same_mailer() concurrently, with the same self:

https://github.com/zopefoundation/zope.sendmail/blob/85e2af8ed2aaf7c31fc0c7a3e7482e99eadedd4e/src/zope/sendmail/tests/test_mailer.py#L126-L146

Line 134 is not thread safe!

https://github.com/zopefoundation/zope.sendmail/blob/85e2af8ed2aaf7c31fc0c7a3e7482e99eadedd4e/src/zope/sendmail/tests/test_mailer.py#L134