search

michalc / sqlite-s3-query

Python functions to query SQLite files stored on S3
MIT License
251 stars 15 forks source link

Incompatibility with httpx 0.23.3 #44

Closed matthewdeanmartin closed 1 year ago

matthewdeanmartin commented 1 year ago

I have a stack trace but haven't reproduced it on a fork yet.

safety and other security tools report the lower versions of httpx as insecure, so I'd like to upgrade to the latest httpx ASAP. As a work around I'm pinned to the 0.18.2 version.

Thanks for your time.

2023-03-14T13:43:20.782-04:00Copy2023-03-14 17:43:20,781 [ERROR] search_service.uwsgi_config: HTTP Error | 2023-03-14 17:43:20,781 [ERROR] search_service.uwsgi_config: HTTP Error
-- | --
  | 2023-03-14T13:43:20.782-04:00 | Traceback (most recent call last):
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/httpx/_client.py", line 871, in stream
  | 2023-03-14T13:43:20.782-04:00 | yield response
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/sqlite_s3_query.py", line 92, in make_auth_request
  | 2023-03-14T13:43:20.782-04:00 | yield response
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/sqlite_s3_query.py", line 143, in get_vfs
  | 2023-03-14T13:43:20.782-04:00 | next(response.iter_bytes())
  | 2023-03-14T13:43:20.782-04:00 | StopIteration
  | 2023-03-14T13:43:20.782-04:00 | The above exception was the direct cause of the following exception:
  | 2023-03-14T13:43:20.782-04:00 | Traceback (most recent call last):
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/flask/app.py", line 1823, in full_dispatch_request
  | 2023-03-14T13:43:20.782-04:00 | rv = self.dispatch_request()
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/flask/app.py", line 1799, in dispatch_request
  | 2023-03-14T13:43:20.782-04:00 | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
  | 2023-03-14T13:43:20.782-04:00 | response = function(request)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
  | 2023-03-14T13:43:20.782-04:00 | response = function(request)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/connexion/decorators/validation.py", line 399, in wrapper
  | 2023-03-14T13:43:20.782-04:00 | return function(request)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/connexion/decorators/parameter.py", line 120, in wrapper
  | 2023-03-14T13:43:20.782-04:00 | return function(**kwargs)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/search_service/controllers/controller_utils.py", line 24, in func_wrapper
  | 2023-03-14T13:43:20.782-04:00 | return func(*args, **kwargs)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/search_service/controllers/cis_api.py", line 64, in does_copyright_pdf_exist
  | 2023-03-14T13:43:20.782-04:00 | query_cis.signed_url_lookup.get_pdf_object_path(copyright_number)
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/search_service/cis_files/query_cis.py", line 161, in get_pdf_object_path
  | 2023-03-14T13:43:20.782-04:00 | results = pool.map(query_year, years)
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/multiprocessing/pool.py", line 367, in map
  | 2023-03-14T13:43:20.782-04:00 | return self._map_async(func, iterable, mapstar, chunksize).get()
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/multiprocessing/pool.py", line 774, in get
  | 2023-03-14T13:43:20.782-04:00 | raise self._value
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/multiprocessing/pool.py", line 125, in worker
  | 2023-03-14T13:43:20.782-04:00 | result = (True, func(*args, **kwds))
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
  | 2023-03-14T13:43:20.782-04:00 | return list(map(*args))
  | 2023-03-14T13:43:20.782-04:00 | ^^^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/search_service/cis_files/query_cis.py", line 143, in query_year
  | 2023-03-14T13:43:20.782-04:00 | with sqlite_s3_query(
  | 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/contextlib.py", line 137, in __enter__
  | 2023-03-14T13:43:20.783-04:00 | return next(self.gen)
  | 2023-03-14T13:43:20.783-04:00 | ^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.783-04:00 | File "/var/lang/lib/python3.11/site-packages/sqlite_s3_query.py", line 383, in sqlite_s3_query
  | 2023-03-14T13:43:20.783-04:00 | with sqlite_s3_query_multi(url,
  | 2023-03-14T13:43:20.783-04:00 | File "/var/lang/lib/python3.11/contextlib.py", line 137, in __enter__
  | 2023-03-14T13:43:20.783-04:00 | return next(self.gen)
  | 2023-03-14T13:43:20.783-04:00 | ^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.783-04:00 | File "/var/lang/lib/python3.11/site-packages/sqlite_s3_query.py", line 359, in sqlite_s3_query_multi
  | 2023-03-14T13:43:20.783-04:00 | with \
  | 2023-03-14T13:43:20.783-04:00 | File "/var/lang/lib/python3.11/contextlib.py", line 137, in __enter__
  | 2023-03-14T13:43:20.783-04:00 | return next(self.gen)
  | 2023-03-14T13:43:20.783-04:00 | ^^^^^^^^^^^^^^
  | 2023-03-14T13:43:20.783-04:00 | RuntimeError: generator raised StopIteration
michalc commented 1 year ago

Ah just looking at the trace:

| 2023-03-14T13:43:20.782-04:00 | File "/var/lang/lib/python3.11/site-packages/sqlite_s3_query.py", line 143, in get_vfs | 2023-03-14T13:43:20.782-04:00 | next(response.iter_bytes())

it looks like you're on an older version of sqlite-s3-query. What was

next(response.iter_bytes())

is now

next(response.iter_bytes(), b'')

Done in https://github.com/michalc/sqlite-s3-query/commit/cb86eae133e1434c72c9df49f4fdc83f5e98871a exactly for more recent httpx versions

matthewdeanmartin commented 1 year ago

Oh, you're right, I was pinned to an earlier version. Thanks!