test_context and test_asyncio crash on Windows 7

[vstinner]

BPO	32641
Nosy	@db3l, @vstinner, @asvetlov, @1st1
PRs	python/cpython#5292

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['3.7', 'expert-asyncio'] title = 'test_context and test_asyncio crash on Windows 7' updated_at = user = 'https://github.com/vstinner' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'yselivanov' components = ['asyncio'] creation = creator = 'vstinner' dependencies = [] files = [] hgrepos = [] issue_num = 32641 keywords = ['patch'] message_count = 11.0 messages = ['310528', '310529', '310530', '310531', '310532', '310538', '310541', '310552', '310554', '310555', '310578'] nosy_count = 4.0 nosy_names = ['db3l', 'vstinner', 'asvetlov', 'yselivanov'] pr_nums = ['5292'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue32641' versions = ['Python 3.7'] ```

[vstinner]

http://buildbot.python.org/all/#/builders/58/builds/498

0:36:25 [252/414/1] test_context crashed (Exit code 3) -- running: test_largefile (239 sec) Windows fatal exception: code 0xc000001d

Current thread 0x00000944 (most recent call first): File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 321 in ctx1_fun File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 339 in test_context_copy1 File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in \_call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1861 in _run_suite File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1951 in run_unittest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in \<module> File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main Fatal Python error: Illegal instruction

Current thread 0x00000944 (most recent call first): File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 321 in ctx1_fun File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 339 in test_context_copy1 File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in \_call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1861 in _run_suite File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1951 in run_unittest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in \<module> File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main

0:46:50 [306/414/2] test_asyncio crashed (Exit code 3) -- running: test_io (487 sec) Windows fatal exception: code 0xc000001d

Current thread 0x00000d24 (most recent call first): File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2191 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2289 in _step File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\events.py", line 88 in _run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 1529 in _run_once File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 391 in run_forever File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 423 in run_until_complete File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2199 in test_context1 File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in \_call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1861 in _run_suite File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1951 in run_unittest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in \<module> File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main Fatal Python error: Illegal instruction

Current thread 0x00000d24 (most recent call first): File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2191 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2289 in _step File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\events.py", line 88 in _run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 1529 in _run_once File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 391 in run_forever File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 423 in run_until_complete File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2199 in test_context1 File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in \_call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1861 in _run_suite File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\init.py", line 1951 in run_unittest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in \<module> File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main

Note: the commit 0bad4d63c654d93e1f32ff35026405a3987db5ca doesn't fix these crashes. (The running build 399 hash this fix, and test_context does still crash.)

[vstinner]

test_asyncio started to crash at build 491:

http://buildbot.python.org/all/#/builders/58/builds/491

This build contains the commit f23746a934177c48eff754411aba54c31d6be2f0: "bpo-32436: Implement PEP-567". I'm not 100% sure that it's the cause of the regression. Since test_context added by this commit also crash, it really smells this the cause of the regression.

[1st1]

It's not a regression, the newly added code doesn't work on some Windows buildbot... looking at it.

[vstinner]

Some hints. Compiler warnings:

..\Python\hamt.c(625): warning C4018: '\<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj] ..\Python\hamt.c(733): warning C4018: '\<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj] ..\Python\hamt.c(939): warning C4018: '\<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj] ..\Python\hamt.c(1122): warning C4018: '\<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]

[1st1]

No, these are trivial uint32_t/Py_SIZE: for (uint32_t i = val_idx + 1; i < Py_SIZE(o); i++); it's something else. I suspect popcount instruction...

[1st1]

I suspect this is a compiler bug. I can't reproduce it on my windows 7 virtual machine and on AppVeyor.

It's not related to HAMT, btw, as there is a multitude of HAMT-specific tests that all pass. The crash is specifically in context.c, and the code there is pretty trivial.

[vstinner]

I suspect this is a compiler bug.

It may be interesting to add the compiler version to test.pythoninfo. It would be simpler to compare Windows buildbots to identify a bug.

Currently, the only info is:

"sys.version: 3.7.0a4+ (heads/master:9d411c1, Jan 23 2018, 15:13:42) [MSC v.1900 32 bit (Intel)]"

Some environment variables are also dumped:

os.environ[VS100COMNTOOLS]: C:\Program Files\Microsoft Visual Studio 10.0\Common7\Tools\ os.environ[VS140COMNTOOLS]: D:\Program Files\Microsoft Visual Studio 14.0\Common7\Tools\ os.environ[VS90COMNTOOLS]: D:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools\

In the compile step, I see lines like:

"Microsoft (R) Build Engine version 14.0.24730.2"

I recall also issues with outdated ucrtbase DLLs. Maybe we can also dump the version of such DLL?

[1st1]

In an off-list conversation with David, it was confirmed that the buildbot in question runs on an old hardware without SSE4.2 support.

I decided to simply stop using native popcount instructions as there's no detectable performance difference when using them vs using the fallback code.

See https://github.com/python/cpython/pull/5292 for details.

[1st1]

New changeset b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab by Yury Selivanov in branch 'master': bpo-32436: Don't use native popcount() (also fixes bpo-32641) (bpo-5292) https://github.com/python/cpython/commit/b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab

[1st1]

I've pushed a commit that should fix the buildbot. Please reopen if it doesn't.

[vstinner]

I've pushed a commit that should fix the buildbot. Please reopen if it doesn't.

test_context and test_asyncio do'nt crash anymore on this buildbot, thanks Yury.

test_asyncio still fail on this buildbot, I created a new issue: bpo-32645, test_asyncio: TLS tests fail on "x86 Windows7" buildbot.