search

onthegomap / planetiler

Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast
Apache License 2.0
1.2k stars 100 forks source link

Structured tag api #895

Closed msbarry closed 1 month ago

msbarry commented 1 month ago

Add an API to features that makes it more convenient to work with feature properties that may be nested lists or maps.

For example:

// for struct from {"a": {"b": {"c": 1}}}
// all return 1:
struct.getStruct("a").get("b").get("c").asInt();
struct.getStruct("a", "b", "c").asInt();
struct.getLong("a.b.c");
// returns null:
struct.getStruct("d", "e", "f").asInt();

You can also treat any string as json and attempt to parse structured attributes from it:

Struct.of("[1,2,3]").get(1).asInt(); // returns 2
Struct.of("""
{"a":{"b":{"c":1}}}
""").get("a", "b", "c").asInt(); // returns 1
Struct.of("""
{"a":{"b":[{"c":1}, {"c":2}]}}
""").get("a.b[].c").get(1) // returns 2

And you can marshal structs into java wrappers 

var struct = Struct.of("""
{
  "name": "place",
  "population": 100,
  "class": "town"
}
""");
record Place(String name, int population, String class) {}
// returns Place[name="place", population=100, class="town"]
struct.as(Place.class);

See StructTest for more examples.

github-actions[bot] commented 1 month ago
This Branch 83e20bcf6a4fae3d432a9fa5e161c649daabae84 Base df19ecf0005df801c51b85564eafd9a19d08d4ce
``` 0:01:07 DEB [archive] - Tile stats: 0:01:07 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (154k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:83k) 2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 4. 10/308/381 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 5. 14/4941/6092 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:64k) 6. 14/4941/6093 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4940/6092 (99k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 8. 11/616/762 (98k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k) 10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k) 0:01:07 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 154 374 443 583 938 339 433 548 773 1.6k 2.1k 7.2k 6.4k 5.8k 4.5k 7.2k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 441 441 441 639 712 1k 1.5k 3.1k 5.6k 3.3k 1.7k 795 936 5.6k landuse 0 0 0 0 548 694 1.6k 6.8k 17k 44k 59k 50k 38k 19k 12k 59k transportation 0 0 0 0 243 782 1.2k 5.9k 8k 24k 17k 19k 65k 48k 33k 65k waterway 0 0 0 0 111 118 0 0 0 3.1k 2.4k 2.1k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1k 3.7k 9.7k 19k 13k 8.2k 4.3k 3.4k 4.4k 19k transportation_name 0 0 0 0 0 0 369 464 1.2k 1.8k 5.4k 4.6k 3.9k 3.4k 18k 18k landcover 0 0 0 0 0 0 0 9.5k 29k 85k 72k 81k 53k 30k 24k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.3k 2.8k 1.4k 1.4k 869 4.3k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 664 327 273 220 220 664 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2.1k 3k 3.4k 2.7k 3.4k poi 0 0 0 0 0 0 0 0 0 0 0 0 501 498 83k 83k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.7k 6k 20k 42k 85k 203k 185k 135k 114k 128k 244k 244k gzipped 6.2k 3.5k 7.1k 5.2k 3.1k 4.8k 14k 29k 60k 149k 138k 98k 83k 91k 154k 154k 0:01:07 DEB [archive] - Max tile: 244k (gzipped: 154k) 0:01:07 DEB [archive] - Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic 0:01:07 DEB [archive] - # tiles: 4,115,012 0:01:07 DEB [archive] - # features: 5,484,355 0:01:07 INF [archive] - Finished in 20s cpu:1m12s avg:3.6 0:01:07 INF [archive] - read 1x(3% 0.5s wait:18s done:1s) 0:01:07 INF [archive] - encode 4x(54% 11s wait:2s) 0:01:07 INF [archive] - write 1x(21% 4s wait:13s) 0:01:07 INF [archive] - Finished in 1m7s cpu:3m38s gc:1s avg:3.3 0:01:07 INF [archive] - FINISHED! 0:01:07 INF [archive] - 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - data errors: 0:01:07 INF [archive] - render_snap_fix_input 16,639 0:01:07 INF [archive] - osm_multipolygon_missing_way 389 0:01:07 INF [archive] - osm_boundary_missing_way 73 0:01:07 INF [archive] - merge_snap_fix_input 12 0:01:07 INF [archive] - osm_boundary_duplicate_member 2 0:01:07 INF [archive] - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:07 INF [archive] - feature_polygon_osm_invalid_multipolygon_empty_after_fix 2 0:01:07 INF [archive] - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:07 INF [archive] - omt_fix_water_before_ne_intersect 1 0:01:07 INF [archive] - feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix 1 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - overall 1m7s cpu:3m38s gc:1s avg:3.3 0:01:07 INF [archive] - lake_centerlines 3s cpu:6s avg:2.1 0:01:07 INF [archive] - read 1x(15% 0.5s done:3s) 0:01:07 INF [archive] - process 4x(0% 0s done:2s) 0:01:07 INF [archive] - write 1x(0% 0s done:2s) 0:01:07 INF [archive] - water_polygons 15s cpu:44s avg:2.8 0:01:07 INF [archive] - read 1x(41% 6s done:8s) 0:01:07 INF [archive] - process 4x(27% 4s wait:4s done:5s) 0:01:07 INF [archive] - write 1x(4% 0.6s wait:10s done:5s) 0:01:07 INF [archive] - natural_earth 7s cpu:13s avg:2 0:01:07 INF [archive] - read 1x(94% 6s) 0:01:07 INF [archive] - process 4x(12% 0.8s wait:6s) 0:01:07 INF [archive] - write 1x(0% 0s wait:6s) 0:01:07 INF [archive] - osm_pass1 2s cpu:7s avg:3.2 0:01:07 INF [archive] - read 1x(2% 0s wait:2s) 0:01:07 INF [archive] - parse 4x(32% 0.6s) 0:01:07 INF [archive] - process 1x(70% 1s) 0:01:07 INF [archive] - osm_pass2 18s cpu:1m12s avg:4 0:01:07 INF [archive] - read 1x(0% 0s wait:11s done:8s) 0:01:07 INF [archive] - process 4x(74% 13s) 0:01:07 INF [archive] - write 1x(3% 0.5s wait:18s) 0:01:07 INF [archive] - ne_lakes 0s cpu:0s avg:0 0:01:07 INF [archive] - boundaries 0s cpu:0s avg:2 0:01:07 INF [archive] - agg_stop 0s cpu:0s avg:11.3 0:01:07 INF [archive] - sort 1s cpu:4s avg:2.5 0:01:07 INF [archive] - worker 1x(50% 0.7s) 0:01:07 INF [archive] - archive 20s cpu:1m12s avg:3.6 0:01:07 INF [archive] - read 1x(3% 0.5s wait:18s done:1s) 0:01:07 INF [archive] - encode 4x(54% 11s wait:2s) 0:01:07 INF [archive] - write 1x(21% 4s wait:13s) 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - archive 108MB 0:01:07 INF [archive] - features 281MB -rw-r--r-- 1 runner docker 84M May 24 14:47 run.jar ``` ``` 0:01:07 DEB [archive] - Tile stats: 0:01:07 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (154k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:83k) 2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 4. 10/308/381 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 5. 14/4941/6092 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:64k) 6. 14/4941/6093 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4940/6092 (99k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 8. 11/616/762 (98k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k) 10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k) 0:01:07 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 154 374 443 583 938 339 433 548 773 1.6k 2.1k 7.2k 6.4k 5.8k 4.5k 7.2k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 441 441 441 639 712 1k 1.5k 3.1k 5.6k 3.3k 1.7k 795 936 5.6k landuse 0 0 0 0 548 694 1.6k 6.8k 17k 44k 59k 50k 38k 19k 12k 59k transportation 0 0 0 0 243 782 1.2k 5.9k 8k 24k 17k 19k 65k 48k 33k 65k waterway 0 0 0 0 111 118 0 0 0 3.1k 2.4k 2.1k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1k 3.7k 9.7k 19k 13k 8.2k 4.3k 3.4k 4.4k 19k transportation_name 0 0 0 0 0 0 369 464 1.2k 1.8k 5.4k 4.6k 3.9k 3.4k 18k 18k landcover 0 0 0 0 0 0 0 9.5k 29k 85k 72k 81k 53k 30k 24k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.3k 2.8k 1.4k 1.4k 869 4.3k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 664 327 273 220 220 664 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2.1k 3k 3.4k 2.7k 3.4k poi 0 0 0 0 0 0 0 0 0 0 0 0 501 498 83k 83k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.7k 6k 20k 42k 85k 203k 185k 135k 114k 128k 244k 244k gzipped 6.2k 3.5k 7.1k 5.2k 3.1k 4.8k 14k 29k 60k 149k 138k 98k 83k 91k 154k 154k 0:01:07 DEB [archive] - Max tile: 244k (gzipped: 154k) 0:01:07 DEB [archive] - Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic 0:01:07 DEB [archive] - # tiles: 4,115,012 0:01:07 DEB [archive] - # features: 5,484,355 0:01:07 INF [archive] - Finished in 20s cpu:1m13s avg:3.7 0:01:07 INF [archive] - read 1x(3% 0.6s wait:18s done:1s) 0:01:07 INF [archive] - encode 4x(54% 11s wait:2s done:1s) 0:01:07 INF [archive] - write 1x(21% 4s wait:13s done:1s) 0:01:07 INF [archive] - Finished in 1m7s cpu:3m41s gc:1s avg:3.3 0:01:07 INF [archive] - FINISHED! 0:01:07 INF [archive] - 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - data errors: 0:01:07 INF [archive] - render_snap_fix_input 16,639 0:01:07 INF [archive] - osm_multipolygon_missing_way 389 0:01:07 INF [archive] - osm_boundary_missing_way 73 0:01:07 INF [archive] - merge_snap_fix_input 12 0:01:07 INF [archive] - osm_boundary_duplicate_member 2 0:01:07 INF [archive] - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:07 INF [archive] - feature_polygon_osm_invalid_multipolygon_empty_after_fix 2 0:01:07 INF [archive] - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:07 INF [archive] - omt_fix_water_before_ne_intersect 1 0:01:07 INF [archive] - feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix 1 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - overall 1m7s cpu:3m41s gc:1s avg:3.3 0:01:07 INF [archive] - lake_centerlines 2s cpu:6s avg:2.4 0:01:07 INF [archive] - read 1x(21% 0.5s done:2s) 0:01:07 INF [archive] - process 4x(0% 0s done:2s) 0:01:07 INF [archive] - write 1x(0% 0s done:2s) 0:01:07 INF [archive] - water_polygons 15s cpu:41s avg:2.7 0:01:07 INF [archive] - read 1x(42% 6s done:7s) 0:01:07 INF [archive] - process 4x(26% 4s wait:4s done:5s) 0:01:07 INF [archive] - write 1x(4% 0.6s wait:9s done:5s) 0:01:07 INF [archive] - natural_earth 6s cpu:13s avg:2 0:01:07 INF [archive] - read 1x(95% 6s) 0:01:07 INF [archive] - process 4x(12% 0.8s wait:6s) 0:01:07 INF [archive] - write 1x(0% 0s wait:6s) 0:01:07 INF [archive] - osm_pass1 2s cpu:6s avg:3.3 0:01:07 INF [archive] - read 1x(2% 0s wait:2s) 0:01:07 INF [archive] - parse 4x(34% 0.6s) 0:01:07 INF [archive] - process 1x(68% 1s) 0:01:07 INF [archive] - osm_pass2 20s cpu:1m17s avg:3.9 0:01:07 INF [archive] - read 1x(0% 0s wait:11s done:9s) 0:01:07 INF [archive] - process 4x(72% 14s) 0:01:07 INF [archive] - write 1x(2% 0.5s wait:19s) 0:01:07 INF [archive] - ne_lakes 0s cpu:0s avg:0 0:01:07 INF [archive] - boundaries 0s cpu:0s avg:2.7 0:01:07 INF [archive] - agg_stop 0s cpu:0s avg:0 0:01:07 INF [archive] - sort 1s cpu:4s avg:2.6 0:01:07 INF [archive] - worker 1x(55% 0.8s) 0:01:07 INF [archive] - archive 20s cpu:1m13s avg:3.7 0:01:07 INF [archive] - read 1x(3% 0.6s wait:18s done:1s) 0:01:07 INF [archive] - encode 4x(54% 11s wait:2s done:1s) 0:01:07 INF [archive] - write 1x(21% 4s wait:13s done:1s) 0:01:07 INF [archive] - ---------------------------------------- 0:01:07 INF [archive] - archive 108MB 0:01:07 INF [archive] - features 281MB -rw-r--r-- 1 runner docker 84M May 24 14:49 run.jar ```

Full logs: https://github.com/onthegomap/planetiler/actions/runs/9225813095

sonarcloud[bot] commented 1 month ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
81.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud