michalc / sqlite-s3-query

Python functions to query SQLite files stored on S3
MIT License
251 stars 15 forks source link

Infinite recursion issue with using latest httpx #42

Closed matthewdeanmartin closed 1 year ago

matthewdeanmartin commented 1 year ago

I honestly can't tell if this is a problem with the SSL packages install on my host, a problem with sqlite-s3-query, or a problem with httpx.

Eventlet got a similar call stack in a bug, but I'm not smart enough to know if it is relevant.

I'm posting this here because I got an error message and a not very good workaround. My host is AWS Linux.

So safety, pip-audit and so on report the lower versions of httpx as insecure. But if I bump to the latest version I get a recursion error.

# my work around...
# This version used to work at 18, but gets flagged as insecure.
# The 23 version creates recursive loops with either sqlite lib
# or it isn't multiprocess safe anymore... not sure.
httpx = "^0.18.2"
response = function(request)
--
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/connexion/decorators/validation.py", line 399, in wrapper
  | 2022-11-09T09:43:24.319-05:00 | return function(request)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/connexion/decorators/parameter.py", line 120, in wrapper
  | 2022-11-09T09:43:24.319-05:00 | return function(**kwargs)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/search_service/controllers/controller_utils.py", line 24, in func_wrapper
  | 2022-11-09T09:43:24.319-05:00 | return func(*args, **kwargs)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/search_service/controllers/cis_api.py", line 64, in does_copyright_pdf_exist
  | 2022-11-09T09:43:24.319-05:00 | query_cis.signed_url_lookup.get_pdf_object_path(copyright_number)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/search_service/cis_files/query_cis.py", line 161, in get_pdf_object_path
  | 2022-11-09T09:43:24.319-05:00 | results = pool.map(query_year, years)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 364, in map
  | 2022-11-09T09:43:24.319-05:00 | return self._map_async(func, iterable, mapstar, chunksize).get()
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 771, in get
  | 2022-11-09T09:43:24.319-05:00 | raise self._value
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 125, in worker
  | 2022-11-09T09:43:24.319-05:00 | result = (True, func(*args, **kwds))
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
  | 2022-11-09T09:43:24.319-05:00 | return list(map(*args))
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/search_service/cis_files/query_cis.py", line 143, in query_year
  | 2022-11-09T09:43:24.319-05:00 | with sqlite_s3_query(
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/contextlib.py", line 119, in __enter__
  | 2022-11-09T09:43:24.319-05:00 | return next(self.gen)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/sqlite_s3_query.py", line 383, in sqlite_s3_query
  | 2022-11-09T09:43:24.319-05:00 | with sqlite_s3_query_multi(url,
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/contextlib.py", line 119, in __enter__
  | 2022-11-09T09:43:24.319-05:00 | return next(self.gen)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/sqlite_s3_query.py", line 360, in sqlite_s3_query_multi
  | 2022-11-09T09:43:24.319-05:00 | get_http_client() as http_client, \
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/sqlite_s3_query.py", line 374, in <lambda>
  | 2022-11-09T09:43:24.319-05:00 | ), get_http_client=lambda: httpx.Client(),
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_client.py", line 673, in __init__
  | 2022-11-09T09:43:24.319-05:00 | self._transport = self._init_transport(
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_client.py", line 721, in _init_transport
  | 2022-11-09T09:43:24.319-05:00 | return HTTPTransport(
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_transports/default.py", line 126, in __init__
  | 2022-11-09T09:43:24.319-05:00 | ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_config.py", line 49, in create_ssl_context
  | 2022-11-09T09:43:24.319-05:00 | return SSLConfig(
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_config.py", line 73, in __init__
  | 2022-11-09T09:43:24.319-05:00 | self.ssl_context = self.load_ssl_context()
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_config.py", line 85, in load_ssl_context
  | 2022-11-09T09:43:24.319-05:00 | return self.load_ssl_context_verify()
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_config.py", line 122, in load_ssl_context_verify
  | 2022-11-09T09:43:24.319-05:00 | context = self._create_default_ssl_context()
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_config.py", line 157, in _create_default_ssl_context
  | 2022-11-09T09:43:24.319-05:00 | set_minimum_tls_version_1_2(context)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/site-packages/httpx/_compat.py", line 30, in set_minimum_tls_version_1_2
  | 2022-11-09T09:43:24.319-05:00 | context.minimum_version = ssl.TLSVersion.TLSv1_2
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/ssl.py", line 587, in minimum_version
  | 2022-11-09T09:43:24.319-05:00 | super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/ssl.py", line 587, in minimum_version
  | 2022-11-09T09:43:24.319-05:00 | super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/ssl.py", line 587, in minimum_version
  | 2022-11-09T09:43:24.319-05:00 | super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  | 2022-11-09T09:43:24.319-05:00 | [Previous line repeated 445 more times]
  | 2022-11-09T09:43:24.319-05:00 | File "/var/lang/lib/python3.9/ssl.py", line 585, in minimum_version
  | 2022-11-09T09:43:24.319-05:00 | if value == TLSVersion.SSLv3:
  | 2022-11-09T09:43:24.319-05:00 | RecursionError: maximum recursion depth exceeded while calling a Python object
michalc commented 1 year ago

Hi @matthewdeanmartin,

Thanks for the report. It's an odd one - nothing leaps to mind as to what could be causing it. The latest set of tests on main ran on v0.23.0 of httpx for example I think

A few questions:

Thanks,

Michal

matthewdeanmartin commented 1 year ago

Shoot, sorry, eventlet is in the mix & it must be doing the monkey patching which makes it hard to see what is going on.

And I can't update to the latest eventlet because gunicorn's maintainer is on hiatus.

This issue can be closed, thanks!

michalc commented 1 year ago

Ah no problem - in fact keep the reports coming, it's nice to see that this is being used.

One workaround has crossed my mind: you can replace the httpx client with one that has a compatible API, so say using python requests. There isn't really documentation on this, but something like this might work:

from contextlib import contextmanager

import requests 
from sqlite_s3_query import sqlite_s3_query

@contextmanager
def get_http_client():
    class Response():
        def __init__(self, response):
            self.response = response
            self.headers = response.headers

        def iter_bytes(self):
            yield from self.response.iter_content()

        def raise_for_status(self):
            return self.response.raise_for_status()

    class Client():
        def __init__(self, session):
            self.session = session

        @contextmanager
        def stream(self, method, url, params, headers):
            yield Response(self.session.request(method, url, params=params, headers=dict(headers)))

    with requests.Session() as session:
        yield Client(session)

with sqlite_s3_query(
    url='https://my-bucket.s3.eu-west-2.amazonaws.com/my-db.sqlite',
    get_http_client=get_http_client,
) as query:
    with query('SELECT * FROM my_table WHERE my_column = ?', params=('my-value',)) as (columns, rows):
        for row in rows:
            print(row)

So this would allow you to keep the most recent httpx installed, so shouldn't get any alerts of vulnerabilities, but not use it, and instead use another http client that work better with the older version of eventlet