ssl: SNI callbacks should not modify context objects

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

BPO	29470
Nosy	@vstinner, @tiran, @FirefighterBlu3

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', '3.7', 'docs'] title = 'ssl: SNI callbacks should not modify context objects' updated_at = user = 'https://github.com/FirefighterBlu3' ``` bugs.python.org fields: ```python activity = actor = 'christian.heimes' assignee = 'docs@python' closed = False closed_date = None closer = None components = ['Documentation'] creation = creator = 'David Ford (FirefighterBlu3)' dependencies = [] files = [] hgrepos = [] issue_num = 29470 keywords = [] message_count = 20.0 messages = ['287198', '287220', '287260', '287262', '287263', '287751', '287761', '287764', '287765', '287811', '287812', '287813', '287845', '287848', '287866', '287875', '287901', '287906', '287913', '301506'] nosy_count = 4.0 nosy_names = ['vstinner', 'christian.heimes', 'docs@python', 'David Ford (FirefighterBlu3)'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue29470' versions = ['Python 2.7', 'Python 3.6', 'Python 3.7'] ```

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

(Only 3.[56] tested)

my application listens on an SSL wrapped port, accepts socket and gets an incoming set of messages. connection time frame is several seconds but due to bad Comcast service frequently we have outages which hang connections. almost every day python crashes as follows:

Feb 07 02:36:11 ranger.blue-labs.org python[11816]: Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in viola Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: \========= Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: \========= Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]

Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]

Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-797 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645) Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-798 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645) Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-794 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645) Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-796 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645) Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-795 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: cutils.cpython-35m-x86_64-linux-gnu.so (deleted)

i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, conditional jump/move on uninitialized value, use of uninitialized value; mostly in libpython3.6m.

[vstinner]

Python 3.6 got a new PYTHONMALLOC env var which should help you ;-)

Please try to get the Python traceback on the crash and use Python builtin memory debugger:

PYTHONMALLOC=debug python3 -X faulthandler ...

i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, (...)

Python memory allocator pymalloc causes false alarms. You can disable pymalloc to use Valgrind using:

PYTHONMALLOC=malloc valgrind python3 ...

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?

[vstinner]

David Ford: "does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?"

Nope, see: http://bugs.python.org/issue18227#msg191610

Please try what I suggested. Buffer overflows may not crash immediatly, but PYTHONMALLOC=debug should help to detect some bugs in memory blocks allocated by Python.

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

yes, it's running now, hasn't crashed yet

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

2017-02-14 06:58:29 INFO connected: mustang.blue-labs.org/107.170.82.162:37188 2017-02-14 06:58:29 INFO No cert required for this host Error in `/usr/bin/python': double free or corruption (out): 0x00007fb0f8003400 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) Fatal Python error: Aborted

Thread 0x00007fb0effff700 (most recent call first): File "/var/bluelabs/python/_cams_threadedserver.py", line 116 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in \_init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/pytho2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) n3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb100de2700 (most recent call first): File "/var/bluelabs/python/_cams_thread2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-6 2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-5 edserver.py", line 116 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in \_init__ File "2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-10 /usr/lib/python3.6/socketserver.py", line 361 in 2017-02-14 06:58:29 INFO disconnect: Thread-11 finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb1025e5700 (most recent call first): File "/var/bluelabs/python/_cams_threadedserver.py", line 137 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in \_init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb102de6700 (most recent call first): File "/var/bluelabs/python/_cams_threadedserver.py", line 137 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in \_init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fb1035e7700 (most recent call first): File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback File "/usr/lib/python3.6/ssl.py", line 683 in do_handshake File "/usr/lib/python3.6/ssl.py", line 1061 in do_handshake File "/var/bluelabs/python/_cams_threadedserver.py", line 152 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in \_init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11131a700 (most recent call first): File "/usr/lib/python3.6/threading.py", line 299 in wait File "/usr/lib/python3.6/threading.py", line 551 in wait File "/var/bluelabs/bots/cams/camsbot.py", line 753 in start_message_queue_handler File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb111b1b700 (most recent call first): File "/var/bluelabs/python/irclib.py", line 230 in process_once File "/var/bluelabs/python/irclib.py", line 246 in process_forever File "/var/bluelabs/python/irclib.py", line 1228 in start File "/var/bluelabs/python/ircbot.py", line 258 in start File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11231c700 (most recent call first): File "/usr/lib/python3.6/selectors.py", line 376 in select File "/usr/lib/python3.6/socketserver.py", line 236 in serve_forever File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11a936400 (most recent call first): File "/usr/lib/python3.6/threading.py", line 1072 in _wait_for_tstate_lock File "/usr/lib/python3.6/threading.py", line 1056 in join File "/usr/lib/python3.6/threading.py", line 1290 in _shutdown Aborted (core dumped)

[vstinner]

You now have to test "PYTHONMALLOC=malloc valgrind python3 ...".

IMHO it's not a bug in Python, but in your code ussing OpenSSL:

File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback

[tiran]

Please install debug symbols and provide a proper C backtrace of the invalid free call. The backtrace in your first comment is missing both line numbers, file and function names.

[tiran]

Please also provide more information about your system: platform, distro, architecture, OpenSSL version, exact Python version etc.

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

in process of creating a non-stripped python install. the default Arch Linux PKGBUILD uses xvfb-run at the end of the build process which is crashing.

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

my SNI callback is a function that chooses which SSL certificate to load and summarized is:

  6 def sni_callback(sock, servername, context):

[...] 44 # load a specific crt/key 45 store = crtstore[servername] 46 47 context.load_cert_chain( certfile = store['certfile'], 48 keyfile = store['keyfile'], 49 password = store['password'] )

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

Arch Linux (updated \~weekly but not often rebooted) on a Digital Ocean droplet.

Kernel 4.5.1-1-ARCH, x86_64, OpenSSL 1.0.2j, Python 3.6.0

[tiran]

Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

yes, i create a context and then when a request comes in, i load the appropriate key. this is how most [all?] SNI capable python server examples show how to do it, or did, at the time I wrote this module a few years ago. from that time, \~py 3.2 until now, it worked perfectly.

thank you for this information, i'll rewrite my callback accordingly. perhaps 18.2.5.3 in https://docs.python.org/3/library/ssl.html can be updated to inform developers how to change their socket.context in an SNI callback

[vstinner]

Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.

Would it be possible to prevent bugs? For example, make SSLContext "read only" while the SNI callback is called? Or my question doesn't make sense? :-) I don't know well how OpenSSL works.

At least, we should warn users in the ssl documentation to mention such crash?

[tiran]

It's not a bug in Python's ssl module. If I understand David's approach correctly, then he is using the SNI callback the wrong way. By using it the wrong way he has discovered a threading bug in OpenSSL. There is some kind of race condition going on in which two threads free and replace the RSA private key at the same time.

I'm -1 to try to make the SSLContext object magically read-only.

David, which examples did you read? The documentation https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_servername_callback is pretty clear:

A typical use of this callback is to change the ssl.SSLSocket‘s SSLSocket.context attribute to a new object of type SSLContext representing a certificate chain that matches the server name.

Apache mod_ssl does it correctly, the first hit on stack overflow, too. https://stackoverflow.com/questions/5113333/how-to-implement-server-name-indication-sni

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

@Christian, that was years ago and there were few examples available of people that had tried to make an SNI capable server. in several cases, people were monkey patching to make a callback. .set_servername_callback() didn't formally show up in ssl mod until 3.4.

i disagree that the documentation is very clear about this. there's one direct reference to usage which is worded as "A typical use of this callback is to...". that doesn't imply 'very clear' :}

i cannot recall anything anywhere that says one must not modify the context after it is used. 18.2.3 implies that a context is a long lived object, longer lived than a socket. it's clear that several people including myself, have mistakenly tried modifying the context after it was built. until now, it worked, probably purely by coincidence.

as a matter of fact, i used to change more than just the key loaded. i changed the ciphers, options, and verify_flags when i knew the incoming IP was incapable of supporting modern ciphers and options. i also modified the verify paths to support a messed up corporate environment.

i suggest the 18.2.3 documentation be modified to be clear about: a) if, when, and how a context can be modified after creation b) an SNI example that changes contexts and explicitly indicates (a)

i don't see any value to not being clear about it, and it's disingenuous to leave nooses hiding in shadows for devs to hang themselves with :-]

[vstinner]

David Ford: would you mind to directly propose a doc patch?

[2b967bd1-2db1-4873-8b2b-6c63ad684fee]

yes, after i've modified my tool and ensured i know the correct way of doing it. one of the problems i'm dealing with is not carting around a global variable holding prebuilt contexts. i find it unappealing.

[tiran]

OK, let's update the documentation with a better example. Are you still interested to contribute a simple example?