LD-GeoJSON abstraction + mini-refactor

[dnomadb]

This includes:

• GeoJSON input abstraction (IE, user need not worry about whether an input is line-delimited geojson or a Feature Collection, or if either comes as stdin or as any number of input files).
• Some test refactoring to put re-used mocks in conftest
• Move to compact json for output (with optional indentation formatting)
• Move informational printing to stderr
• Removal of unneeded file handlers / print handlers
• Raising when requests 404 / otherwise error

I'll also need to update some of the docs but would like a prelim review before starting in on that.

cc @mapsam

[dnomadb]

I have the cligj functionality working; for validate-source and add-source the CLI is agnostic to the flavor of the GeoJSON feature set; we can do:

Path to line-delimited geojson:

$ tilesets validate-source tests/fixtures/valid.ldgeojson
Validating features
✔ valid

Or a path to a feature collection:

$ tilesets validate-source $(dst=/tmp/fc.geojson; cat tests/fixtures/valid.ldgeojson | fio collect > $dst; echo $dst)
Validating features
✔ valid

Either can be read from stdin:

$ cat tests/fixtures/valid.ldgeojson | fio collect | tilesets validate-source
Validating features
✔ valid
$ cat tests/fixtures/valid.ldgeojson | tilesets validate-source
Validating features
✔ valid

Which means that if a user has a shapefile they can use the fiona cli to convert to GeoJSON and stream in on-the-fly:

$ fio cat $(dst=/tmp/f.shp; cat tests/fixtures/valid.ldgeojson | fio collect | fio load -f "ESRI Shapefile" --dst-crs EPSG:4326 $dst; echo $dst) \
| tilesets validate-source
Validating features
✔ valid

@mapsam while doing this I cleaned up a bunch of the functionality for these subcommands:

• Raise when erroring + remove manual exiting https://github.com/mapbox/tilesets-cli/pull/22/files#diff-d3bb9f9ead05989b2f5de0a637b1ebc2R327
• Remove manual json error messaging
• Logging only in CLI + formatting utils: https://github.com/mapbox/tilesets-cli/pull/22/files#diff-164c18f2544285a3a97e8703275c2f04R30
• A little more standard re-org (tilesets/scripts/utils.py to tilesets/utils.py and tilesets/cli.py to tilesets/scripts/cli.py

Should I get the rest of the functions in sync with these patterns now as well?

[mapsam]

@dnomadb nice!

Should I get the rest of the functions in sync with these patterns now as well?

👍 go for it - since this PR is already doing some cleaning, might as well clean it all!

[dnomadb]

I tested this out, and everything seems to work well. Here's what I did:

First, I used fiona to create a feature stream from a shapefile, and passed it into add-source:

$ fio cat /Users/damonburgett/Downloads/q403shp/gunit.shp \
| tilesets add-source dnomadb tacoma-geology

And:

$ tilesets view-source dnomadb tacoma-geology
{"id": "mapbox://tileset-source/dnomadb/tacoma-geology", "files": 1, "size": 4049925, "size_nice": "3.86MB"}

Then I created and validated a recipe:

$ tilesets validate-recipe /tmp/recipe.json
{"valid": true}

After which I created a tileset with this recipe:

$ tilesets create dnomadb.tilesets-test-0 -r /tmp/recipe.json -n tacoma-quad-geo

And then published it:

$ tilesets publish dnomadb.tilesets-test-

After I while I checked it:

$ tilesets status dnomadb.tilesets-test-0
{"id": "dnomadb.tilesets-test-0", "queued": 0, "processing": 0, "success": 0, "failed": 1, "last_completed_job": "ck16q31ng001w01qn26hgc4fr"}

O no! It errored! So I checked the jobs:

$ tilesets jobs dnomadb.tilesets-test-0
[{"id":"ck16q31ng001w01qn26hgc4fr","stage":"failed","created":1569866673436,"created_nice":"Mon Sep 30 2019 18:04:33 GMT+0000 (UTC)","published":1569866673435,"tilesetId":"dnomadb.tilesets-test-0","errors":["Error processing tileset dnomadb.tilesets-test-0, job ck16q31ng001w01qn26hgc4fr. Please contact support."],"warnings":[]}]

I realized that I didn't project the data before streaming it, so I deleted this source:

$ tilesets delete-source dnomadb tacoma-geology
"Source deleted.

Perhaps in the future this can have a are you sure? prompt. Then I re-added it with the correct projection and with less precision (6 places):

$ fio cat /Users/damonburgett/Downloads/q403shp/gunit.shp --dst-crs EPSG:4326 --precision 6 \
| tilesets add-source dnomadb tacoma-geology

Then, re-published the tileset:

$ tilesets publish dnomadb.tilesets-test-0
{"message": "Processing dnomadb.tilesets-test-0.", "jobId": "ck16s5xrn000i01p97sqo5khn"}
You can view the status of your tileset with the `tilesets status dnomadb.tilesets-test-0` command.
```bash
$ tilesets status dnomadb.tilesets-test-0
{"id": "dnomadb.tilesets-test-0", "queued": 0, "processing": 0, "success": 1, "failed": 1, "last_completed_job": "ck16r7q0m000201s02swub71p"}

Hooray! I checked the jobs:

$ tilesets jobs dnomadb.tilesets-test-0
[{"id": "ck16q31ng001w01qn26hgc4fr", "stage": "failed", "created": 1569866673436, "created_nice": "Mon Sep 30 2019 18:04:33 GMT+0000 (UTC)", "published": 1569866673435, "tilesetId": "dnomadb.tilesets-test-0", "errors": ["Error processing tileset dnomadb.tilesets-test-0, job ck16q31ng001w01qn26hgc4fr. Please contact support."], "warnings": []}, {"id": "ck16r7q0m000201s02swub71p", "stage": "success", "created": 1569868571254, "created_nice": "Mon Sep 30 2019 18:36:11 GMT+0000 (UTC)", "published": 1569868571252, "tilesetId": "dnomadb.tilesets-test-0", "errors": [], "warnings": [], "layer_stats": {"gunit": {"capped": 0, "minzoom": 10, "zooms": {"10": {"capped": 0, "ymin": 358, "avg_size": 26782.88888888889, "tiles": 9, "xmin": 162, "xmax": 164, "max_size": 50059, "sum_area": 6358.975434990324, "sum_size": 241046, "ymax": 360, "size_histogram": [4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, "11": {"capped": 0, "ymin": 716, "avg_size": 10570.866666666667, "tiles": 30, "xmin": 324, "xmax": 329, "max_size": 23550, "sum_area": 5287.205285049536, "sum_size": 317126, "ymax": 720, "size_histogram": [30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, "12": {"capped": 0, "ymin": 1432, "avg_size": 3990.5925925925926, "tiles": 108, "xmin": 648, "xmax": 659, "max_size": 10156, "sum_area": 4753.118190878719, "sum_size": 430984, "ymax": 1440, "size_histogram": [108, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, "13": {"capped": 0, "ymin": 2864, "avg_size": 1548.229468599034, "tiles": 414, "xmin": 1297, "xmax": 1319, "max_size": 4660, "sum_area": 4555.071599592087, "sum_size": 640967, "ymax": 2881, "size_histogram": [414, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, "14": {"capped": 0, "ymin": 5729, "avg_size": 665.8049872122762, "tiles": 1564, "xmin": 2594, "xmax": 2639, "max_size": 2281, "sum_area": 4302.007507382449, "sum_size": 1041319, "ymax": 5762, "size_histogram": [1564, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}}, "polygon_count": 17701, "total_tiles": 2125, "maxzoom": 14, "point_count": 0, "linestring_count": 0}}}, {"id": "ck16s5xrn000i01p97sqo5khn", "stage": "queued", "created": 1569870167603, "created_nice": "Mon Sep 30 2019 19:02:47 GMT+0000 (UTC)", "published": 1569870167602, "tilesetId": "dnomadb.tilesets-test-0", "errors": [], "warnings": []}]

A few notes / ideas for the future

• We should prompt for deletion; I'll ticket this
• I knew how to debug this, but perhaps we should give more error context? This isn't a CLI issue per-se but is worthwhile thinking about. We could have also caught this in the CLI which might be worthwhile.