Create fixture suite for new 2.0 spec validations

mapsam commented 8 years ago

Background

This came up in a recent 2.0 spec update of mapnik-upload-validate where @springmeyer encountered errors that weren't being thrown after we had removed the deprecated vt.parse() method. Adding the check back in helped, but we are still in a spot where "our existing test coverage appears to be completely missing for this code".

The question now is do we write tests for invalid tiles within mapnik-upload-validate or do we ensure all of these potential failures have been properly tested UPSTREAM in mapnik-vector-tile?

Action

This ticket is a reminder to put together a suite of invalid vector tiles so we can test within mapnik-vector-tile to safely ensure downstream tests in other modules depending on node-mapnik are not re-testing the same exact stuff.

Tasks

We'll want to cover all of the validation errors in vector_tile_is_valid.hpp:

[x] Vector Tile message has two or more layers with the same name
[x] Vector Tile message has an unknown tag
[x] Vector Tile Layer message has no name
[x] Vector Tile Layer message has no extent
[x] Vector Tile Layer message has no features
[x] Vector Tile Layer message has an invalid version
[x] Vector Tile Layer message has an unknown tag
[x] Vector Tile Value message contains more then one values
[x] Vector Tile Value message contains no values
[x] Vector Tile Value message has an unknown tag
[x] Vector Tile Feature message has no geometry
[x] Vector Tile Feature message has multiple geometries
[x] Vector Tile Feature message is missing a geometry type
[x] Vector Tile Feature message has an invalid geometry type
[x] Vector Tile Feature message has an unknown tag
[ ] Buffer is not encoded as a valid PBF
[ ] UNKNOWN ERROR

We'll also need a solid workflow in place to actually create these tiles.

Related question: what is our current status for properly formatted tiles and tests written around them?

mapsam commented 8 years ago

A number of these cases are covered by evilmvt. I will work on creating the others today!

mapsam commented 8 years ago

@flippmoke I've created MVT buffers for every case above and more that we can start using in Mapnik Vector Tile! https://github.com/mapbox/evilmvt

mapsam commented 8 years ago

Update: Now at https://github.com/mapbox/mvt-fixtures which includes valid MVT fixtures as well. I'll start working on including these into the library!

pnorman commented 8 years ago

I recommend testing that any fields in the protobuf lie within the numeric range of the fields of the spec, i.e. that it's not writing to fields not defined in https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto

springmeyer commented 7 years ago

Work on a new and better fixture suite is now taking place at https://github.com/mapbox/mvt-fixtures/issues/15. Once that is done, we'll integrate back here. Creating a new ticket to track that integration -> #255

mapbox / mapnik-vector-tile