search

python / cpython

The Python programming language
https://www.python.org
Other
63.07k stars 30.2k forks source link

int(s), float(s) and others may cause segmentation fault #78268

Closed 6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 closed 6 years ago

6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago
BPO 34087
Nosy @vstinner, @ezio-melotti, @fenrrir, @methane, @jonashaag, @serhiy-storchaka, @miss-islington
PRs
  • python/cpython#8274
  • python/cpython#8279
  • python/cpython#8280
    		•
    Files
  • testproj.tar.gz
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', '3.7', '3.8', 'type-crash'] title = 'int(s), float(s) and others may cause segmentation fault' updated_at = user = 'https://github.com/fenrrir' ``` bugs.python.org fields: ```python activity = actor = 'methane' assignee = 'none' closed = True closed_date = closer = 'methane' components = ['Interpreter Core'] creation = creator = 'fenrrir' dependencies = [] files = ['47688'] hgrepos = [] issue_num = 34087 keywords = ['patch'] message_count = 34.0 messages = ['321394', '321404', '321406', '321407', '321450', '321452', '321491', '321533', '321560', '321561', '321566', '321567', '321577', '321578', '321581', '321586', '321588', '321589', '321590', '321591', '321593', '321594', '321596', '321602', '321603', '321605', '321613', '321616', '321617', '321618', '321621', '321641', '321642', '321645'] nosy_count = 7.0 nosy_names = ['vstinner', 'ezio.melotti', 'fenrrir', 'methane', 'jonash', 'serhiy.storchaka', 'miss-islington'] pr_nums = ['8274', '8279', '8280'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue34087' versions = ['Python 3.7', 'Python 3.8'] ```

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago
    methane commented 6 years ago

    It's difficult to investigate without reproducible example.

    Do you use 3rd party extension module? Typical cause of these crash is bug in 3rd party extensions.

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    Sorry for that. I’m not able to make a little example to reproduce this bug. It’s happens during Django tests on a very large code base. A interest thing is that not happens with “-X dev” parameters. Please, any suggestions how I can get more details about this bug?

    methane commented 6 years ago

    If example cannot be small, you can share the project on Github or other place. If your project is private, you can create OSS version of reproducer. If you can't create OSS reproducer, we can't debug it.

    3rd party library list may helpful, but I'm not sure.

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    Running with -X faulthandler

    Fatal Python error: Segmentation fault

    Current thread 0x00007fff89cf2380 (most recent call first): File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 526 in resolve File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 500 in resolve File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 113 in _get_response File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/views.py", line 43 in do_revisionview File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/middleware.py", line 51 in \_call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 81 in getresponse File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 138 in \_call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 467 in request File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 404 in generic File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 332 in get File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 517 in get File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 615 in run File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 663 in __call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/testcases.py", line 206 in __call File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 122 in run File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 84 in __call File "/Users/rodrigo/root/lib/python3.7/unittest/runner.py", line 176 in run File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 569 in run_suite File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 603 in run_tests File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 59 in handle File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 335 in execute File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 288 in run_from_argv File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 26 in run_from_argv File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init.py", line 365 in execute File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init__.py", line 371 in execute_from_command_line File "manage.py", line 10 in \<module> Segmentation fault: 11

    The Django code:

        def resolve(self, path):
        path = str(path)  # path may be a reverse_lazy object
        tried = []
        match = self.pattern.match(path)
        if match:
            new_path, args, kwargs = match
            for pattern in self.url_patterns:
                try:
                    sub_match = pattern.resolve(new_path)
                except Resolver404 as e:
                    sub_tried = e.args[0].get('tried')
                    if sub_tried is not None:
                        tried.extend([pattern] + t for t in sub_tried)
                    else:
                        tried.append([pattern])
                else:
                    if sub_match:
                        # Merge captured arguments in match with submatch
                        sub_match_dict = dict(kwargs, **self.default_kwargs)
                        # Update the sub_match_dict with the kwargs from the sub_match.
                        sub_match_dict.update(sub_match.kwargs)
                        # If there are *any* named groups, ignore all non-named groups.
                        # Otherwise, pass all non-named arguments as positional arguments.
                        sub_match_args = sub_match.args
                        if not sub_match_dict:
                            sub_match_args = args + sub_match.args
                        return ResolverMatch(
                            sub_match.func,
                            sub_match_args,
                            sub_match_dict,
                            sub_match.url_name,
                            [self.app_name] + sub_match.app_names,
                            [self.namespace] + sub_match.namespaces,
                        )
                    tried.append([pattern]) # Segfault happens here!
            raise Resolver404({'tried': tried, 'path': new_path})
        raise Resolver404({'path': path})

    According to faulthandler the segmentation fault happens on tried.append([pattern]).

    vstinner commented 6 years ago

    Please try to set PYTHONMALLOC=debug environment variable and run again your test. I hope that Python will be able to catch a memory corruption somewhere. If it's the case, you can try:

    PYTHONMALLOC=debug python3 -X tracemalloc=10 (...)

    To see where the corrupted object has been allocated.

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    Unfortunately with 'PYTHONMALLOC=debug' the segmentation fault do not happen.

    methane commented 6 years ago

    Would you provide list of extension modules? What happens when removing some of them?

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    I also have a segfault that goes away with malloc debugging. Not sure if it's the same issue.

    My extension modules are

    venv/lib/python3.7/site-packages//_yaml.cpython-37m-darwin.so venv/lib/python3.7/site-packages//netifaces.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_webp.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingft.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingcms.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imaging.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingtk.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmorph.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/builder.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/_elementpath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/diff.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/clean.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/etree.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/objectify.cpython-37m-darwin.so venv/lib/python3.7/site-packages//coverage/tracer.cpython-37m-darwin.so

    Unfortunately I can't test the application without any of these, but maybe we can match my list with fenrrir's.

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    Btw my segfault is from Django too, but that may just be a coincidence

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    Extensions:

    ./_cffi_backend.cpython-37m-darwin.so ./_mssql.cpython-37m-darwin.so ./_yaml.cpython-37m-darwin.so ./Crypto/Cipher/_AES.cpython-37m-darwin.so ./Crypto/Cipher/_ARC2.cpython-37m-darwin.so ./Crypto/Cipher/_ARC4.cpython-37m-darwin.so ./Crypto/Cipher/_Blowfish.cpython-37m-darwin.so ./Crypto/Cipher/_CAST.cpython-37m-darwin.so ./Crypto/Cipher/_DES.cpython-37m-darwin.so ./Crypto/Cipher/_DES3.cpython-37m-darwin.so ./Crypto/Cipher/_XOR.cpython-37m-darwin.so ./Crypto/Hash/_MD2.cpython-37m-darwin.so ./Crypto/Hash/_MD4.cpython-37m-darwin.so ./Crypto/Hash/_RIPEMD160.cpython-37m-darwin.so ./Crypto/Hash/_SHA224.cpython-37m-darwin.so ./Crypto/Hash/_SHA256.cpython-37m-darwin.so ./Crypto/Hash/_SHA384.cpython-37m-darwin.so ./Crypto/Hash/_SHA512.cpython-37m-darwin.so ./Crypto/PublicKey/_fastmath.cpython-37m-darwin.so ./Crypto/Util/_counter.cpython-37m-darwin.so ./Crypto/Util/strxor.cpython-37m-darwin.so ./cryptography/hazmat/bindings/_constant_time.abi3.so ./cryptography/hazmat/bindings/_openssl.abi3.so ./cryptography/hazmat/bindings/_padding.abi3.so ./Cython/Compiler/Code.cpython-37m-darwin.so ./Cython/Compiler/FlowControl.cpython-37m-darwin.so ./Cython/Compiler/FusedNode.cpython-37m-darwin.so ./Cython/Compiler/Lexicon.cpython-37m-darwin.so ./Cython/Compiler/Parsing.cpython-37m-darwin.so ./Cython/Compiler/Pythran.cpython-37m-darwin.so ./Cython/Compiler/Scanning.cpython-37m-darwin.so ./Cython/Compiler/Visitor.cpython-37m-darwin.so ./Cython/Plex/Actions.cpython-37m-darwin.so ./Cython/Plex/Scanners.cpython-37m-darwin.so ./Cython/Runtime/refnanny.cpython-37m-darwin.so ./Cython/StringIOTree.cpython-37m-darwin.so ./Cython/Tempita/_tempita.cpython-37m-darwin.so ./lazy_object_proxy/cext.cpython-37m-darwin.so ./lxml/_elementpath.cpython-37m-darwin.so ./lxml/builder.cpython-37m-darwin.so ./lxml/etree.cpython-37m-darwin.so ./lxml/html/clean.cpython-37m-darwin.so ./lxml/html/diff.cpython-37m-darwin.so ./lxml/objectify.cpython-37m-darwin.so ./markupsafe/_speedups.cpython-37m-darwin.so ./PIL/_imaging.cpython-37m-darwin.so ./PIL/_imagingcms.cpython-37m-darwin.so ./PIL/_imagingmath.cpython-37m-darwin.so ./PIL/_imagingmorph.cpython-37m-darwin.so ./PIL/_imagingtk.cpython-37m-darwin.so ./psycopg2/_psycopg.cpython-37m-darwin.so ./pymssql.cpython-37m-darwin.so ./reportlab/graphics/_renderPM.cpython-37m-darwin.so ./reportlab/lib/_rl_accel.cpython-37m-darwin.so ./setproctitle.cpython-37m-darwin.so ./simplejson/_speedups.cpython-37m-darwin.so ./wrapt/_wrappers.cpython-37m-darwin.so

    requirements.txt

    alabaster==0.7.11 amqp==2.2.2 appnope==0.1.0 arrow==0.12.0 asn1crypto==0.24.0 astroid==1.6.5 Babel==2.6.0 beautifulsoup4==4.6.0 billiard==3.5.0.3 boto3==1.7.24 botocore==1.10.24 celery==4.1.0 certifi==2018.4.16 cffi==1.11.2 chardet==3.0.4 colorama==0.3.9 colorful==0.4.0 configparser==3.5.0 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.2.2 cssselect==1.0.3 Cython==0.28.2 decorator==4.3.0 dj-database-url==0.4.2 Django==2.0.6 django-braces==1.12.0 django-celery-beat==1.1.1 django-celery-results==1.0.1 django-ckeditor==5.2.1 django-colorful==1.2 django-crispy-forms==1.6.1 django-extensions==1.9.8 django-extra-views==0.9.0 django-filter==1.0.4 django-filters==0.2.1 django-formset-js==0.5.0 django-formtools==2.1 django-fsm==2.6.0 django-jquery-js==3.1.1 django-js-asset==1.1.0 django-localflavor==1.6.2 django-model-utils==3.1.1 django-mptt==0.9.0 django-pagination==1.0.10 django-querysetsequence==0.8 django-recaptcha2==1.0.3 django-rest-swagger==2.1.2 django-reversion==2.0.13 django-reversion-compare==0.8.4 django-storages==1.6.6 django-tables2==1.17.1 django-tables2-reports==0.1.3 django-taggit==0.22.1 django-wkhtmltopdf==3.1.0 djangorestframework==3.7.3 djangorestframework-filters==0.10.2 djangorestframework-jwt==1.11.0 docutils==0.14 feedparser==5.2.1 google-api-python-client==1.6.4 gunicorn==19.7.1 html5lib==1.0.1 httplib2==0.10.3 idna==2.6 imagesize==0.7.1 ipaddress==1.0.19 ipdb==0.10.3 ipython==6.2.1 ipython-genutils==0.2.0 isort==4.2.15 itypes==1.1.0 jedi==0.12.1 Jinja2==2.10 jmespath==0.9.3 kombu==4.2.1 lazy-object-proxy==1.3.1 ldap3==2.4 lxml==4.1.1 MarkupSafe==1.0 mccabe==0.6.1 MechanicalSoup==0.10.0 minio==4.0.0 model-mommy==1.5.0 oauth2client==4.1.2 olefile==0.45.1 openapi-codec==1.3.2 parso==0.3.1 pathlib2==2.3.0 pbr==3.1.1 pdfrw==0.4 pexpect==4.3.1 pickleshare==0.7.4 Pillow==4.3.0 prompt-toolkit==1.0.15 psycopg2==2.7.3.2 ptyprocess==0.5.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycparser==2.18 pycpfcnpj==1.2 pycrypto==2.6.1 Pygments==2.2.0 PyJWT==1.6.4 pylint==1.8.1 pyOpenSSL==17.5.0 PyPDF2==1.26.0 pyquery==1.3.0 python-dateutil==2.6.1 python-logstash==0.4.6 python-magic==0.4.15 python-memcached==1.58 pytz==2018.4 PyYAML==3.13 qrcode==5.3 raven==6.4.0 reportlab==3.4.0 requests==2.18.4 rsa==3.4.2 s3transfer==0.1.13 setproctitle==1.1.10 simplegeneric==0.8.1 simplejson==3.13.2 six==1.11.0 snowballstemmer==1.2.1 Sphinx==1.6.5 sphinxcontrib-websupport==1.1.0 suds-py3==1.3.3.0 traitlets==4.3.2 typing==3.6.4 Unidecode==0.4.21 uritemplate==3.0.0 urllib3==1.22 vine==1.1.4 wcwidth==0.1.7 webencodings==0.5.1 wrapt==1.10.11 xlrd==1.1.0 xlutils==2.0.0 xlwt==1.3.0 xmljson==0.1.9

    methane commented 6 years ago

    Hmm, all segfault are happend on macOS?

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    I can reproduce this on Ubuntu 18.04.

    INADA, I have a full gdb backtrace with Python 3.7 development build. I'd like to share it with you privately as I'm concerned it may contain sensible information. I know that's a bit unconventional; if you have other suggestions I'm happy to follow along.

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    I did remove PyYAML, lxml and Pillow here but segfault still happen

    methane commented 6 years ago

    @jonash, Thank you. Would you share core file too? You can get core file when reproducing it after ulimit -c unlimited.

    My mail address is "songofacandy [at] gmail.com". Please share it via Dropbox or Google Drive.

    methane commented 6 years ago

    I received two core files ("core" and "core.23") from fenrrir, thank you. But I can't investigate well, sorry. I think someone breaks memory, and SEGV happened later. I need to reproduce it on my machine...

    In core.23, SEGV happens in kill() syscall. I can't get how it can happened.

    (gdb) bt 10

    0 0x00007ffff6c10187 in kill () at ../sysdeps/unix/syscall-template.S:78

    1 0x0000555555778f71 in os_kill_impl (module=\<optimized out>, signal=\<optimized out>, pid=\<optimized out>) at ../Modules/posixmodule.c:6749

    2 os_kill (module=\<optimized out>, args=\<optimized out>, nargs=\<optimized out>) at ../Modules/clinic/posixmodule.c.h:2672

    3 0x00005555555dd4ba in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=140737353236984, args=0x7ffff7ee7e60, self=\<module at remote 0x7ffff7f221d8>,

    method=0x555555c007a0 \<posix_methods+1920\>) at ../Objects/call.c:651

    In "core", SEGV happens in pymalloc. Pool is broken and I suppose some buffer overflow breaks the pool, but I'm not sure.

    #0  pymalloc_alloc (ptr_p=ptr_p@entry=0x7fffee81a640, nbytes=40, ctx=<optimized out>) at ../Objects/obmalloc.c:1398
#1  0x000055555562ed78 in pymalloc_alloc (ctx=<optimized out>, nbytes=40, ptr_p=0x7fffee81a640) at ../Objects/obmalloc.c:1565
#2  _PyObject_Malloc (ctx=<optimized out>, nbytes=40) at ../Objects/obmalloc.c:1555
#3  0x000055555561d5d5 in new_dict_with_shared_keys (keys=0x7ffff4e4c2d0) at ../Objects/dictobject.c:607
#4  _PyObjectDict_SetItem (tp=tp@entry=0x555555e8b338, dictptr=0x7fffedfe4250, key=key@entry='s', value=value@entry='\n  <h3>') at ../Objects/dictobject.c:4311
#5  0x000055555562b2d4 in _PyObject_GenericSetAttrWithDict (dict=0x0, value='\n  <h3>', name='s', obj=<TextNode at remote 0x7fffedfe4240>) at ../Objects/object.c:1319
#6  PyObject_GenericSetAttr (obj=<TextNode at remote 0x7fffedfe4240>, name='s', value='\n  <h3>') at ../Objects/object.c:1341
#7  0x000055555562a09f in PyObject_SetAttr (v=v@entry=<TextNode at remote 0x7fffedfe4240>, name=<optimized out>, value=value@entry='\n  <h3>') at ../Objects/object.c:978
#8  0x00005555555c67d8 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:2002
#9  0x00005555555c1787 in function_code_fastcall (co=co@entry=0x7ffff4e90930, args=<optimized out>, args@entry=0x7fffee81a930, nargs=2,
    globals=globals@entry={'__name__': 'django.template.base', '__doc__': "\nThis is the Django template system...
../Objects/call.c:283

    (gdb) p pool $1 = (poolp) 0x7fffee861000 (gdb) p bp $2 = (block *) 0x7a672f \<error: Cannot access memory at address 0x7a672f> (gdb) p *pool $3 = {ref = {_padding = 0x2c \<error: Cannot access memory at address 0x2c>, count = 44}, freeblock = 0x7a672f \<error: Cannot access memory at address 0x7a672f>, nextpool = 0x555555ba1fd0 \<usedpools+48>, prevpool = 0x555555ba1fd0 \<usedpools+48>, arenaindex = 170, szidx = 4, nextoffset = 1768, maxnextoffset = 4056} (gdb) p *(pool->nextpool) $4 = {ref = {_padding = 0x7fffeec2a000 "}", count = 4005732352}, freeblock = 0x7fffeea8e000 "\006", nextpool = 0x7fffee861000, prevpool = 0x7fffee861000, arenaindex = 4004200448, szidx = 32767, nextoffset = 4004200448, maxnextoffset = 32767}

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    Reduced it to something that seems unicode related?

    No extension modules involved. Vanilla Django project with a single url + template.

    See testproj/urls.py and tmpl/index.html

    vstinner commented 6 years ago

    If someone expects progress on these issues, we need to get a repository that we can clone to reproduce the bug. Then the code must be simplified as much as possible to reduce the code base. The best is when you still trigger the crash with less than 1000 lines of code and no or very few dependencies.

    methane commented 6 years ago

    @jonash Thanks! Would you give me concrete step to run the project and reproduce the segfault?

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    Sure.

    Unpack archive, create new 3.7 venv with Django (latest version is fine), ./manage.py runserver, curl localhost:8000.

    methane commented 6 years ago

    I tried it but I can't reproduce...

    methane commented 6 years ago

    @jonash What Python binary do you use?

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    Here's a Docker image that reproduces the bug.

    FROM ubuntu:18.04 RUN apt update && apt install -y python3.7-dbg python3.7-venv python3-venv wget RUN python3.7 -m venv venv RUN venv/bin/pip install django RUN wget https://bugs.python.org/file47688/testproj.tar.gz -O - | tar xfz - CMD cd /testproj && /venv/bin/python manage.py runserver & sleep 5; wget -t1 localhost:8000 >/dev/null 2>/dev/null

    Of course this also works outside Docker. I have reproduced with macOS 10.13.5 (17F77) and Ubuntu 18.04 (Docker).

    On macOS it's Python 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)] on darwin, installed from Homebrew.

    On Ubuntu it's whatever is in the 18.04 repositories.

    6f3c3f7c-83f3-4be4-8c43-34121d20d8f5 commented 6 years ago

    I can reproduce the segmentation fault using 'testproj.tar.gz' with homebrew and compiled from source. MacOS X High Sierra 10.13.5 (17F77).

    methane commented 6 years ago

    Very thanks! I found I set export PYTHONMALLOC=pymalloc_debug. That's why I can't reproduce it.

    Now I can reproduce and I get where memory is broken, and understood what cause:

    Python traceback is:
Traceback (most recent call first):
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 767, in __init__
    self.literal = float(var)
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 646, in __init__
    var_obj = Variable(constant).resolve({})
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 568, in compile_filter
    return FilterExpression(token, self)
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/templatetags/i18n.py", line 368, in do_translate
    message_string = parser.compile_filter(bits[1])
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse
    compiled_result = compile_func(self, token)
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loader_tags.py", line 233, in do_block
    nodelist = parser.parse(('endblock',))
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse
    compiled_result = compile_func(self, token)
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 198, in compile_nodelist
    return parser.parse()
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 160, in __init__
    self.nodelist = self.compile_nodelist()
  File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loaders/base.py", line 30, in get_template

    C traceback is:

    #0  0x0000557880378c95 in _Py_string_to_number_with_underscores (s=0x7ff07d8e7bc0 "\"So erstellen Sie einen Zahlungslink f?ded_blocks'", orig_len=39, what=0x55788046d60a "float",
    obj='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"', arg=0x7ff07d9368b0, innerfunc=0x557880276b10 <float_from_string_inner>) at Python/pystrtod.c:410
#1  0x0000557880279b23 in PyFloat_FromString (v='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"') at Objects/floatobject.c:208
#2  0x0000557880279cd5 in float_new_impl (x=<optimized out>, type=0x5578806e4ee0 <PyFloat_Type>) at Objects/floatobject.c:1627
#3  float_new (type=<optimized out>, args=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwargs=<optimized out>) at Objects/clinic/floatobject.c.h:181
#4  0x00005578802cfdc5 in type_call (type=type@entry=0x5578806e4ee0 <PyFloat_Type>, args=args@entry=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwds=kwds@entry=0x0)
    at Objects/typeobject.c:929
#5  0x000055788026b3bf in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x5578806e4ee0>, stack=<optimized out>, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0)
    at Objects/call.c:199
#6  0x00005578802526f4 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at Python/ceval.c:4600
#7  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3183
#8  0x000055788024fb17 in function_code_fastcall (co=co@entry=0x7ff07ec4f0c0, args=<optimized out>, args@entry=0x7ffc786f2330, nargs=2,
    globals=globals@entry={'__name__': 'django.template.base', '__doc__': "\nThis is the Django template system.\n\nHow it works:\n\nThe Lexer.tokenize() function converts a template string
(i.e., a string containing\nmarkup with custom template tags) to tokens, which can be either plain text\n(TOKEN_TEXT), variables (TOKEN_VAR) or block statements (TOKEN_BLOCK).\n\nThe Parser(
) class takes a list of tokens in its constructor, and its parse()\nmethod returns a compiled template -- which is, under the hood, a list of\nNode objects.\n\nEach Node is responsible for c
reating some sort of output -- e.g. simple text\n(TextNode), variable values in a given context (VariableNode), results of basic\nlogic (IfNode), results of looping (ForNode), or anything el
se. The core Node\ntypes are TextNode, VariableNode, IfNode and ForNode, but plugin modules can\ndefine their own custom node types.\n\nEach Node has a render() method, which takes a Context
 and returns a string of\nthe rendered node. For example, the render() method of a Variable Node ret...(truncated)) at Objects/call.c:283
#9  0x000055788026ad87 in _PyFunction_FastCallDict (func=<function at remote 0x7ff07ec2f1e0>, args=0x7ffc786f2330, nargs=<optimized out>, kwargs=0x0) at Objects/call.c:322
#10 0x000055788026ddc1 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffc786f2330, callable=<function at remote 0x7ff07ec2f1e0>) at Objects/call.c:98
    vstinner commented 6 years ago

    It might be a duplicate of bpo-33954: bug in _PyUnicode_InsertThousandsGrouping().

    serhiy-storchaka commented 6 years ago

    Is it possible to write a reproducer in Python?

    The string returned by _PyUnicode_TransformDecimalAndSpaceToASCII() is still a NUL-terminated, but the NUL can be far past the end of the string.

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes.

    vstinner commented 6 years ago

    I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes.

    We just have to call float() with a non-ASCII string, no? An assertion must fail without the fix when Python is compiled in debug mode.

    ef46bb13-8e88-4488-a20f-75e542f6f274 commented 6 years ago

    The assertion in the patched code, yes. The segfault in the unpatched code, no.

    methane commented 6 years ago

    When I commented out the fix:

    $ ./python -c 'int("こんにちは")'
python: Objects/unicodeobject.c:484: _PyUnicode_CheckConsistency: Assertion `PyUnicode_READ(kind, data, ascii->length) == 0' failed.
Aborted

    I added this to test cases.

    methane commented 6 years ago

    New changeset 16dfca4d829e45f36e71bf43f83226659ce49315 by INADA Naoki in branch 'master': bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274) https://github.com/python/cpython/commit/16dfca4d829e45f36e71bf43f83226659ce49315

    miss-islington commented 6 years ago

    New changeset c721472fb83d1f7c7606bcf33ba2d42d6127a764 by Miss Islington (bot) in branch '3.7': bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274) https://github.com/python/cpython/commit/c721472fb83d1f7c7606bcf33ba2d42d6127a764

    methane commented 6 years ago

    New changeset b2f8aa0c998d331ab2b4c701756a6427c0e91d48 by INADA Naoki in branch '3.6': bpo-34087: Backport tests for int/float/complex (GH-8274) https://github.com/python/cpython/commit/b2f8aa0c998d331ab2b4c701756a6427c0e91d48