openstates / openstates-geo

Source for state legislative district map tiles for openstates.org
MIT License
24 stars 12 forks source link

Add CI, to auto-build and -deploy updated tiles #8

Closed mileswwatkins closed 5 years ago

mileswwatkins commented 7 years ago

Sort of a pain to do locally, since the job takes a long time and lots of DLs

mileswwatkins commented 5 years ago

I've Dockerized, but running within Docker Compose results in a crash within the ogr2ogr line. This appears to be related to hardware resources, and happens even when I've allocated 12 GB RAM and 2 CPU cores to Docker Machine. It's likely that the 4 GB RAM limit within CircleCI won't pass this threshold, but I'm going to check this.

make-tiles_1  | ./make-tiles.sh: line 65:     8 Killed                  ogr2ogr -clipsrc ./data/cb_2017_us_nation_5m.shp -where "GEOID NOT LIKE '%ZZZ'" -f GeoJSON ./data/sld-clipped.geojson ./data/sld.geojson
mileswwatkins commented 5 years ago

Bleh, yeah, I get this in the CircleCI output:

./make-tiles.sh: line 65:   370 Killed                  ogr2ogr -clipsrc ./data/cb_2017_us_nation_5m.shp -where "GEOID NOT LIKE '%ZZZ'" -f GeoJSON ./data/sld-clipped.geojson ./data/sld.geojson
Exited with code 137

Hint: Exit code 137 typically means the process is killed because it was running out of memory
Hint: Check if you can optimize the memory usage in your app
Hint: Max memory usage of this container is 4287582208

I can take a hint ;)

https://circleci.com/gh/openstates/openstates-district-maps/12

mileswwatkins commented 5 years ago

Nice! My refactor to run the ogr2ogr clipping on individual states instead of on the country as a whole has resulted in a pipeline that'll run within the 4 GB RAM max on CI.

https://circleci.com/gh/openstates/openstates-district-maps/13

cc https://github.com/openstates/openstates-district-maps/commit/e7583a388b03834d432bd3e3dd61e1adba55eafa

mileswwatkins commented 5 years ago

Only remaining bit is to get CircleCI to pass the Mapbox environment variables to its "remote Docker environment." Otherwise, everything's running successfully:

https://circleci.com/gh/openstates/openstates-district-maps/26

mileswwatkins commented 5 years ago

Whabam