search

cenpy-devs / cenpy

Explore and download data from Census APIs
Other
183 stars 44 forks source link

Create persistent HTTP connection for REST connections #140

Open ronnie-llamado opened 3 years ago

ronnie-llamado commented 3 years ago

Currently a new connection is opened on every HTTP interaction (see log output below). Introducing a requests.session tied to APIConnection and TigerConnection objects may show some performance gains, especially on larger transactions.

The main difference would amount to changing from requests.get to self.session.get (inside APIConnection and TigerConnection).

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.census.gov:443
DEBUG:urllib3.connectionpool:https://api.census.gov:443 "GET /data/2017/acs/acs5/variables.json HTTP/1.1" 200 664592
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.census.gov:443
DEBUG:urllib3.connectionpool:https://api.census.gov:443 "GET /data/2017/acs/acs5/geography.json HTTP/1.1" 200 1274
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.census.gov:443
DEBUG:urllib3.connectionpool:https://api.census.gov:443 "GET /data/2017/acs/acs5/tags.json HTTP/1.1" 200 312
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.census.gov:443
DEBUG:urllib3.connectionpool:https://api.census.gov:443 "GET /data/2017/acs/acs5/examples.json HTTP/1.1" 200 1946
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): tigerweb.geo.census.gov:443
DEBUG:urllib3.connectionpool:https://tigerweb.geo.census.gov:443 "GET /arcgis/rest/services/TIGERweb?f=json HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): tigerweb.geo.census.gov:443
DEBUG:urllib3.connectionpool:https://tigerweb.geo.census.gov:443 "GET /arcgis/rest/services/TIGERweb/tigerWMS_ACS2017/MapServer?f=json HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): tigerweb.geo.census.gov:443
DEBUG:urllib3.connectionpool:https://tigerweb.geo.census.gov:443 "GET /arcgis/rest/services/TIGERweb/tigerWMS_ACS2017/MapServer/layers?f=json HTTP/1.1" 200 None

If done, this will require some refactoring of remote.py and tiger.py. Perhaps even a BaseClass that can be shared between the two.

jGaboardi commented 3 years ago

@ronnie-llamado Is this failure related?

ronnie-llamado commented 3 years ago

@ronnie-llamado Is this failure related?

I don't think so. That failure looks like it originates server side since it's an HTTP error 504. It might be worth rerunning the build (especially since ubuntu python 3.9 build passed).

jGaboardi commented 3 years ago

Yeah, it passes in the PR, but has failed in two reruns on merge. Probably will clear up on its own.

ronnie-llamado commented 3 years ago

After triggering a re-run once more, it looks like it just passed. One of those things that comes with testing off a live API.