Some initial timings suggest we can gain speed by using s3 URL scheme instead of current HTTPS:
In [8]: %timeit -n 10 pd.read_json("https://openaq-fetches.s3.amazonaws.com/realtime/2013-11-26/2013-11-26.ndjson",
...: lines=True)
311 ms ± 37.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [9]: %timeit -n 10 pd.read_json("s3://openaq-fetches/realtime/2013-11-26/2013-11-26.ndjson", lines=True)
159 ms ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Some initial timings suggest we can gain speed by using
s3
URL scheme instead of current HTTPS:similar to #113
Originally posted by @zmoon in https://github.com/noaa-oar-arl/monetio/issues/116#issuecomment-1585078076