SIGSEGV (Address boundary error)

[c8b83f72-b9f7-49f0-aed4-903cf0978a3f]

BPO	34355
Nosy	@methane, @matrixise, @zhangyangyu, @JulienPalard, @yohanboniface
Superseder	bpo-34087: int(s), float(s) and others may cause segmentation fault

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['3.7', 'type-crash'] title = 'SIGSEGV (Address boundary error)' updated_at = user = 'https://github.com/yohanboniface' ``` bugs.python.org fields: ```python activity = actor = 'methane' assignee = 'none' closed = True closed_date = closer = 'methane' components = [] creation = creator = 'ybon' dependencies = [] files = [] hgrepos = [] issue_num = 34355 keywords = [] message_count = 8.0 messages = ['323263', '323267', '323270', '323271', '323272', '323304', '324048', '324169'] nosy_count = 5.0 nosy_names = ['methane', 'matrixise', 'xiang.zhang', 'mdk', 'ybon'] pr_nums = [] priority = 'normal' resolution = 'duplicate' stage = 'resolved' status = 'closed' superseder = '34087' type = 'crash' url = 'https://bugs.python.org/issue34355' versions = ['Python 3.7'] ```

[c8b83f72-b9f7-49f0-aed4-903cf0978a3f]

Hi!

Just installed 3.7 (ArchLinux) and I've a SIGSEGV on one of my projects. I've a hard time reducing to a minimal testcase, because it seems whatever random piece of code I remove the crash disappears at some point.

Here is the repository:

https://framagit.org/ybon/trefle

To reproduce, install the project in a 3.7 venv with python setup.py develop then run python trefle/bin.py (or even python -c 'from trefle import routine').

Here is the output I have:

Initializing config
Done initializing config
fish: “python trefle/bin.py” terminated by signal SIGSEGV (Address boundary error)

Here are some elements:

• if I run the code with PYTHONMALLOC=debug, I have no crash
• the project is using quite a lot of unicode (French content written in config files), even in some file names
• the project is using asyncio (but it does not seem directly related at first look)
• it is running without issue as is on python 3.6

Here is a gdb backtrace:

$ gdb python                                                                                                                                GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run trefle/bin.py 
Starting program: /home/ybon/.virtualenvs/trefle/bin/python trefle/bin.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Initializing config
Done initializing config

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff791a9ce in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
(gdb) backtrace 
#0  0x00007ffff791a9ce in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
#1  0x00007ffff79fec6e in ?? () from /usr/lib/libpython3.7m.so.1.0
#2  0x00007ffff7a05874 in PyParser_ASTFromStringObject () from /usr/lib/libpython3.7m.so.1.0
#3  0x00007ffff7a693f2 in Py_CompileStringObject () from /usr/lib/libpython3.7m.so.1.0
#4  0x00007ffff7a695c3 in ?? () from /usr/lib/libpython3.7m.so.1.0
#5  0x00007ffff795963f in _PyMethodDef_RawFastCallDict () from /usr/lib/libpython3.7m.so.1.0
#6  0x00007ffff79597d1 in _PyCFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#7  0x00007ffff79f7e16 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#8  0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#9  0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#10 0x00007ffff79f3142 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#11 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#12 0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#13 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#14 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#15 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#16 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#17 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#18 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#19 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#20 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#21 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#22 0x00007ffff793a08b in _PyFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#23 0x00007ffff7949888 in ?? () from /usr/lib/libpython3.7m.so.1.0
#24 0x00007ffff79b71b9 in _PyObject_CallMethodIdObjArgs () from /usr/lib/libpython3.7m.so.1.0
#25 0x00007ffff792e285 in PyImport_ImportModuleLevelObject () from /usr/lib/libpython3.7m.so.1.0
#26 0x00007ffff79f4434 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#27 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#28 0x00007ffff7939f34 in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#29 0x00007ffff7939f5c in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#30 0x00007ffff7a05a64 in ?? () from /usr/lib/libpython3.7m.so.1.0
#31 0x00007ffff7959709 in _PyMethodDef_RawFastCallDict () from /usr/lib/libpython3.7m.so.1.0
#32 0x00007ffff79597d1 in _PyCFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#33 0x00007ffff79f7e16 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#34 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#35 0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#36 0x00007ffff79f6933 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#37 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#38 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#39 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#40 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#41 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#42 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#43 0x00007ffff793a08b in _PyFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#44 0x00007ffff7949888 in ?? () from /usr/lib/libpython3.7m.so.1.0
#45 0x00007ffff79b71b9 in _PyObject_CallMethodIdObjArgs () from /usr/lib/libpython3.7m.so.1.0
#46 0x00007ffff792e285 in PyImport_ImportModuleLevelObject () from /usr/lib/libpython3.7m.so.1.0
#47 0x00007ffff79f4434 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#48 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#49 0x00007ffff7939f34 in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#50 0x00007ffff7939f5c in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#51 0x00007ffff7a68770 in ?? () from /usr/lib/libpython3.7m.so.1.0
#52 0x00007ffff7a6a54a in PyRun_FileExFlags () from /usr/lib/libpython3.7m.so.1.0
#53 0x00007ffff7a6bac5 in PyRun_SimpleFileExFlags () from /usr/lib/libpython3.7m.so.1.0
#54 0x00007ffff7a6da8f in ?? () from /usr/lib/libpython3.7m.so.1.0
#55 0x00007ffff7a6e420 in _Py_UnixMain () from /usr/lib/libpython3.7m.so.1.0
#56 0x00007ffff7dc9003 in __libc_start_main () from /usr/lib/libc.so.6
#57 0x000055555555477a in _start ()

Thanks for your help on tracking this! :)

Yohan

[JulienPalard]

Can reproduce with python3.7 from Debian packages, but can't reproduce with a python3.7 built with --with-pydebug.

[matrixise]

With the last revision of 3.7 (w/o --with-debug), I don't get this issue on Fedora 28 :/

[matrixise]

and what's the issue with asyncio ?

[c8b83f72-b9f7-49f0-aed4-903cf0978a3f]

Thanks all :)

As noted by Julien, to reproduce the test cases, one also needs to install the dev requirements (or just pip instal minicli hupper):

pip install -r requirements-dev.txt

and what's the issue with asyncio ?

Nothing specific as far as I can tell. I mentioned it because not all projects use asyncio so I thought it was an significative point to have in mind, just in case. Also I blindly checked "asyncio" in the "Components" because the project does use asyncio, without foreseeing that this would point asyncio as a guilty and that asyncio maintainers would have been specifically CCed. Thanks for fixing this.

One other thing to notice just in case: the code base do have unicode chars (in comments, strings and raw strings).

[JulienPalard]

After noticing that without pydebug I can reproduce in v3.7.0 but not in master I ran a git bisect, the following commit looks like it fixes the issue:

commit 16dfca4d829e45f36e71bf43f83226659ce49315
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Sat Jul 14 12:06:43 2018 +0900

    bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274)

    `_PyUnicode_TransformDecimalAndSpaceToASCII()` missed trailing NUL char.
    It caused buffer overflow in `_Py_string_to_number_with_underscores()`.

    This bug is introduced in 9b6c60cb.

[zhangyangyu]

So this is a dupliate of bpo-34087 and we can close it?

[methane]

I think so.