psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.2k stars 9.34k forks source link

Requests v2.32.0 caused the error `Segmentation fault` when including the `cert` parameter #6745

Closed mingshuang closed 4 months ago

mingshuang commented 5 months ago

After updating the requests to v2.32.0, my service started failure and raised the error Segmentation fault. After some debugging and including faulthandler.enable() into my code, I saw the trace is below, and no such as kind problem with version v2.31.0

File "/opt/vpnaas/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 418 in ssl_wrap_socket
  File "/opt/vpnaas/lib64/python3.11/site-packages/urllib3/connection.py", line 419 in connect
  File "/opt/vpnaas/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 1058 in _validate_conn
  File "/opt/vpnaas/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 404 in _make_request
  File "/opt/vpnaas/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 715 in urlopen
  File "/opt/vpnaas/lib64/python3.11/site-packages/requests/adapters.py", line 564 in send
  File "/opt/vpnaas/lib64/python3.11/site-packages/requests/sessions.py", line 703 in send
  File "/opt/vpnaas/lib64/python3.11/site-packages/requests/sessions.py", line 589 in request
  File "/opt/vpnaas/lib64/python3.11/site-packages/requests/api.py", line 59 in request
  File "/opt/vpnaas/lib64/python3.11/site-packages/requests/api.py", line 73 in get

Expected Result

There is no Segmentation fault

Actual Result

Saw the Segmentation fault randomly.

Reproduction Steps

To help to reproduce the problem, I wrote a script below to simulate my service code. and before executing the script, a certificate is needed and can be generated with command openssl req -x509 -newkey rsa:4096 -keyout client.key -out client.crt -days 365 -nodes

import concurrent.futures
import random
import uuid
from threading import Thread
from time import time

import requests

def do_request():
    start = time()
    random_id = uuid.uuid4()
    delay = random.randint(1, 5)
    print("start {} delay {} seconds".format(random_id, delay))
    endpoints = []
    endpoints.append('https://httpbin.org/delay/' + str(delay))
    delay = str(random.randint(1, 5)) + 's'
    endpoints.append('https://run.mocky.io/v3/0432e9f0-674f-45bd-9c18-628b861c2258?mocky-delay=' + str(delay))
    random.shuffle(endpoints)
    response = None
    for endpoint in endpoints:
        try:
            print("start {} delay {} seconds".format(random_id, endpoint))
            if 'run' in endpoint:
                cert = './client.crt', './client.key'
                response = requests.get(endpoint, timeout=random.randint(1, 5), cert=cert)
            else:
                response = requests.get(endpoint, timeout=random.randint(1, 5))
        except Exception as e:
            print(e)
    end = time()

    print("finished {} in {} seconds".format(random_id, end - start))
    return response

def measure():
    cnt = 20
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        futures = []
        for server in range(1, cnt):
            futures.append(executor.submit(do_request))
        for future in concurrent.futures.as_completed(futures):
            pass

for i in range(1, 500):
    threads = [Thread(target=measure, args=()) for _ in range(5)]

    for t in threads: t.start()
    for t in threads: t.join()

System Information

$ python -m requests.help
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.7"
  },
  "platform": {
    "release": "4.18.0-553.5.1.el8_10.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.0"
  },
  "system_ssl": {
    "version": "101010bf"
  },
  "urllib3": {
    "version": "1.26.18"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
nateprewitt commented 4 months ago

@mingshuang there were initially some issues with the first releases of 2.32.x which is why 2.32.0 and 2.32.1 are yanked from PyPI. Can you please try this with 2.32.3 (the latest release) and if the issue is still reproducing there, check this patch https://github.com/psf/requests/pull/6731/files?

nateprewitt commented 4 months ago

Resolving as a duplicate #6731 since we haven't heard back.

mingshuang commented 4 months ago

hi @nateprewitt yes, I tried 2.32.3 and the result is same. and also tried the PR https://github.com/psf/requests/pull/6731, this PR indeed address my problem.