Use httpx or requests but not both

[simonw]

I want to standardize on httpx - so I need to figure out streaming. https://www.python-httpx.org/quickstart/#streaming-responses

I want the equivalent of this:

https://github.com/simonw/big-local-datasette/blob/48708336a6fe8d80b3295d5c76132a56cc545290/populate_tables.py#L12-L15

[simonw]

Probably this:

def url_to_dicts(url):
    with httpx.stream("GET", url) as response:
        reader = csv.DictReader(response.iter_lines())
        yield from reader

[simonw]

I couldn't get httpx streaming working without any decode errors, so I'll go with requests instead.

Fetching UHJvamVjdDo3NDQ0YTU5NC01MzlkLTQyZjktYTI3My0yMWE2MWRmMjE5MDE=: 2018_county_census.csv
2018_county_census 504921
Traceback (most recent call last):
  File "populate_tables.py", line 70, in <module>
    populate_tables(db)
  File "populate_tables.py", line 65, in populate_tables
    db[table_name].insert_all(url_to_dicts(url))
  File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 994, in insert_all
    chunk = list(chunk)
  File "populate_tables.py", line 15, in url_to_dicts
    yield from reader
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py", line 112, in __next__
    row = next(self.reader)
  File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/httpx/_models.py", line 931, in iter_lines
    for text in self.iter_text():
  File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/httpx/_models.py", line 926, in iter_text
    yield decoder.decode(chunk)
  File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/httpx/_decoders.py", line 196, in decode
    raise DecodingError() from None
httpx._exceptions.DecodingError