ua-snap / data-api

SNAP data API
MIT License
4 stars 0 forks source link

New demographics fields + improved CSV #491

Closed Joshdpaul closed 1 day ago

Joshdpaul commented 1 week ago

This PR fetches the new fields added to the demographics data during this PR, limits API returns to communities with >=50 people, and improves the CSV output by tranposing the table, adding more detailed descrioptions of the data elements, and re-ordering the fields to match the display order in NCR.

closes #490 closes #489 closes #487 closes #482 closes #476 closes #475

To test:

Nabesna should not return data as JSON or CSV

http://127.0.0.1:5000/demographics/AK256 http://127.0.0.1:5000/demographics/AK256?format=csv



The descriptive text in the CSV output has grown considerably here, and so could use some careful proofreading to make sure the sentences are sane and reference the actual datasets that the particular line of data came from. Crosschecking with the `README.md` would be much appreciated!

**Note:** In order to use `gs.earthmaps.io` in this branch, I had to bypass SSL certificate verification in the route-specific `demographics.validate_community_id()` function and also in more widely used `fetch_data.make_get_request()` function. I didn't know how else to get this to work, but I imagine we need to change this in production. Any thoughts on how & when is best to re-introduce the SSL verification would be welcomed! 
Joshdpaul commented 1 week ago

@cstephen Thank you for the detailed review! Great job catching all those little things, they are important!

I was able to address all your comments, and took a stab at a 403 template. Not sure if that's the most appropriate code, or if the message should be something else, but we can adjust that pretty easily in future work.

I am still unable to make requests without turning off SSL verification. I get this error:

SSLError
requests.exceptions.SSLError: HTTPSConnectionPool(host='gs.earthmaps.io', port=443): Max retries exceeded with url: /geoserver/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=demographics:demographics&outputFormat=application/json&propertyName=(id) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1135)')))