search

azavea / pfb-network-connectivity

PFB Bicycle Network Connectivity
Other
40 stars 10 forks source link

Handle NULL values in city_fips column in batch shapefiles #832

Closed KlaasH closed 3 years ago

KlaasH commented 3 years ago

Overview

In the method that processes batch shapefile uploads, which was updated in PR #814 to look for neighborhoods based on their unique attributes then update their other attributes based on the shapefile, having some neighborhoods with city_fips values and some without caused a crash, because the ones without returned None as their city_fips value, which violates the null constraint of the field.

This adds a check to convert None values into empty strings.

Resolves #831

Testing Instructions

Here's a batch shapefile with one town that has a FIPS code: WI_and_IA_amesfips.zip The FIPS is 1916900, so if you add a line with that code to your city_fips_speed.csv file (download it from s3://pfb-public-documents/city_fips_speed.csv if you don't already have one) and run the analysis for Ames, IA, (per the command logged by the batch process), you should see it pick up a city speed limit. Also, in the admin neighborhood list, the other neighborhoods shouldn't have FIPS codes but Ames, IA, should.

Checklist

kevinearldenny commented 3 years ago

@KlaasH I'm working to test this PR now. I have done the following steps:

I'm not entirely sure what you mean by:

(per the command logged by the batch process)

I'm currently trying to run: NB_OUTPUT_DIR=/data/output/ames-test \ ./scripts/run-local-analysis \ PFB_JOB_ID=3fe5516c-5b3c-4039-a4ce-62160a80c45b /Users/kevindenny/Downloads/WI_and_IA_amesfips.zip ia 19 but am encountering an error here. Can you expand a bit on the correct instructions to test this?

KlaasH commented 3 years ago

Oops, yeah, I skipped a step. Go to http://localhost:9301/#/admin/analysis-jobs/create/batch/ and use that zip file to submit an analysis batch. The log messages from your scripts/server should show a series of run-local-analysis commands printed by the django_q_1 task. One of them (probably the 3rd) will include ames.zip in the boundary file path. Running that one in the VM should pick up the city speed limit.

kevinearldenny commented 3 years ago

The speed limit that I entered for Ames is copied to the analysis result, and Ames is the only neighborhood with a FIPS code - looks good 👍