search

fiboa / specification

Field Boundaries for Agriculture (fiboa) - a specification that describes important properties of field boundaries
Apache License 2.0
9 stars 3 forks source link

Define FlatGeoBuf encoding #24

Open m-mohr opened 5 months ago

StefanBrand commented 4 months ago

Some related resources:

StefanBrand commented 3 months ago

Here are some experiments based on the Austrian GSA dataset.

Tools

Steps

  1. Convert to FlatGeobuf: 
    ogr2ogr GSA_Austria_2023-2_sample.fgb INSPIRE_SCHLAEGE_2023-2_POLYGON.gpkg \
  -limit 2 \
  -mo 'fiboa={"fiboa_version": "0.2.0", "fiboa_extensions": []}' \
  -nomd \
  -nln parcels
  2. Find length of Header: hexdump --skip 8 -n 4 -d GSA_Austria_2023-2_sample.fgb -> 2084
  3. Dump header into file: dd skip=12 count=2084 bs=1 if=GSA_Austria_2023-2_sample.fgb > header.bin
  4. Deserialize into JSON: flatc --json --strict-json --raw-binary src/fbs/header.fbs -- header.bin

Result

This is the resulting JSON:

{
  "name": "parcels",
  "envelope": [
    672737.349999999977,
    476317.109999999986,
    675804.430000000051,
    477193.350000000035
  ],
  "geometry_type": "Polygon",
  "columns": [
    {
      "name": "FS_KENNUNG",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "SNAR_BEZEICHNUNG",
      "type": "String",
      "width": 240
    },
    {
      "name": "SL_FLAECHE_BRUTTO_HA",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "GEO_ID",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "INSPIRE_ID",
      "type": "String",
      "width": 500
    },
    {
      "name": "GML_ID",
      "type": "String",
      "width": 500
    },
    {
      "name": "GML_IDENTIFIER",
      "type": "String",
      "width": 500
    },
    {
      "name": "SNAR_CODE",
      "type": "String",
      "width": 30
    },
    {
      "name": "GEO_PART_KEY",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "LOG_PKEY",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "GEOM_DATE_CREATED",
      "type": "DateTime",
      "width": 0
    },
    {
      "name": "FART_ID",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "GEO_TYPE",
      "type": "String",
      "width": 50
    },
    {
      "name": "GML_GEOM",
      "type": "Binary",
      "width": 0
    },
    {
      "name": "GML_LENGTH",
      "type": "Double",
      "precision": 0
    },
    {
      "name": "KZ_BIO_OEPUL_JN",
      "type": "String",
      "width": 1
    }
  ],
  "features_count": 2,
  "crs": {
    "org": "EPSG",
    "code": 31287,
    "name": "MGI / Austria Lambert",
    "wkt": "PROJCRS[\"MGI / Austria Lambert\",BASEGEOGCRS[\"MGI\",DATUM[\"Militar-Geographische Institut\",ELLIPSOID[\"Bessel 1841\",6377397.155,299.1528128,LENGTHUNIT[\"metre\",1]]],PRIMEM[\"Greenwich\",0,ANGLEUNIT[\"degree\",0.0174532925199433]],ID[\"EPSG\",4312]],CONVERSION[\"Austria Lambert\",METHOD[\"Lambert Conic Conformal (2SP)\",ID[\"EPSG\",9802]],PARAMETER[\"Latitude of false origin\",47.5,ANGLEUNIT[\"degree\",0.0174532925199433],ID[\"EPSG\",8821]],PARAMETER[\"Longitude of false origin\",13.3333333333333,ANGLEUNIT[\"degree\",0.0174532925199433],ID[\"EPSG\",8822]],PARAMETER[\"Latitude of 1st standard parallel\",49,ANGLEUNIT[\"degree\",0.0174532925199433],ID[\"EPSG\",8823]],PARAMETER[\"Latitude of 2nd standard parallel\",46,ANGLEUNIT[\"degree\",0.0174532925199433],ID[\"EPSG\",8824]],PARAMETER[\"Easting at false origin\",400000,LENGTHUNIT[\"metre\",1],ID[\"EPSG\",8826]],PARAMETER[\"Northing at false origin\",400000,LENGTHUNIT[\"metre\",1],ID[\"EPSG\",8827]]],CS[Cartesian,2],AXIS[\"northing (X)\",north,ORDER[1],LENGTHUNIT[\"metre\",1]],AXIS[\"easting (Y)\",east,ORDER[2],LENGTHUNIT[\"metre\",1]],USAGE[SCOPE[\"Topographic mapping (medium and small scale).\"],AREA[\"Austria.\"],BBOX[46.4,9.53,49.02,17.17]],ID[\"EPSG\",31287]]"
  },
  "metadata": "{\"fiboa\":\"{\\\"fiboa_version\\\": \\\"0.2.0\\\", \\\"fiboa_extensions\\\": []}\"}"
}

Note that OGR already seems to quote the JSON, which results in too many quoted characters.

m-mohr commented 3 months ago

Thanks for the experiments. That doesn't look ideal with regards to GDAL... It should result in valid JSON, otherwise it feels pretty useless.

m-mohr commented 3 months ago

I think we should open an issue in the GDAL issue tracker and see what Even says... Anyway, it shouldn't block this encoding. Alternatively, we can upgrade the fiboa CLI.

StefanBrand commented 3 months ago

I was just about to open an issue, but had second thoughts. I mean, GDAL's escaping of strings for metadata is intended behaviour. Please go ahead and investigate with Even yourself.

m-mohr commented 3 months ago

I'm not interested in GDAL support for adding fiboa metadata to FlatGeoBuf at this point so I'll not investigate further. I was more thinking about whether a way to add JSON to metadata is on the roadmap. Currently it seems to support only scalars via GDAL it seems.

StefanBrand commented 3 months ago

Another experiment to read a feature (same setup as in https://github.com/fiboa/specification/issues/24#issuecomment-2161099119:

  1. I made an educated guess based on Flatgeobuf: Implementer's Guide that the index would be 120 bytes (3 nodes `a 40 bytes)
  2. hexdump --skip 2216 -n 4 -d GSA_Austria_2023-2_sample.fgb -> 700
  3. dd bs=1 skip=2220 count=700 if=GSA_Austria_2023-2_sample.fgb > feature.bin
  4. flatc --json --strict-json --raw-binary src/fbs/feature.fbs -- feature.bin

Result

JSON output

```json { "geometry": { "xy": [ 675535.709999999963, 477193.350000000035, 675529.570000000065, 477180.640000000014, 675804.119999999995, 477040.450000000012, 675804.430000000051, 477056.130000000005, 675770.209999999963, 477073.609999999986, 675764.940000000061, 477072.190000000002, 675763.849999999977, 477071.880000000005, 675761.570000000065, 477072.559999999998, 675760.790000000037, 477074.679999999993, 675761.800000000047, 477077.900000000023, 675535.709999999963, 477193.350000000035 ] }, "properties": [ 0, 0, 0, 0, 0, 88, 222, 148, 152, 65, 1, 0, 11, 0, 0, 0, 71, 82, 195, 156, 78, 66, 82, 65, 67, 72, 69, 2, 0, 73, 123, 106, 146, 171, 90, 219, 63, 3, 0, 0, 0, 0, 196, 179, 188, 153, 65, 4, 0, 113, 0, 0, 0, 104, 116, 116, 112, 115, 58, 47, 47, 100, 97, 116, 97, 46, 105, 110, 115, 112, 105, 114, 101, 46, 103, 118, 46, 97, 116, 47, 48, 48, 57, 53, 47, 55, 49, 55, 53, 100, 50, 97, 102, 45, 100, 49, 56, 56, 45, 52, 101, 99, 50, 45, 98, 48, 55, 57, 45, 100, 52, 99, 48, 101, 53, 52, 49, 55, 102, 55, 55, 47, 101, 108, 117, 46, 69, 120, 105, 115, 116, 105, 110, 103, 76, 97, 110, 100, 85, 115, 101, 79, 98, 106, 101, 99, 116, 47, 49, 48, 55, 57, 52, 57, 50, 57, 55, 47, 77, 70, 65, 45, 50, 48, 50, 51, 5, 0, 89, 0, 0, 0, 65, 84, 46, 48, 48, 57, 53, 46, 55, 49, 55, 53, 100, 50, 97, 102, 45, 100, 49, 56, 56, 45, 52, 101, 99, 50, 45, 98, 48, 55, 57, 45, 100, 52, 99, 48, 101, 53, 52, 49, 55, 102, 55, 55, 46, 101, 108, 117, 46, 69, 120, 105, 115, 116, 105, 110, 103, 76, 97, 110, 100, 85, 115, 101, 79, 98, 106, 101, 99, 116, 46, 49, 48, 55, 57, 52, 57, 50, 57, 55, 46, 77, 70, 65, 45, 50, 48, 50, 51, 6, 0, 113, 0, 0, 0, 104, 116, 116, 112, 115, 58, 47, 47, 100, 97, 116, 97, 46, 105, 110, 115, 112, 105, 114, 101, 46, 103, 118, 46, 97, 116, 47, 48, 48, 57, 53, 47, 55, 49, 55, 53, 100, 50, 97, 102, 45, 100, 49, 56, 56, 45, 52, 101, 99, 50, 45, 98, 48, 55, 57, 45, 100, 52, 99, 48, 101, 53, 52, 49, 55, 102, 55, 55, 47, 101, 108, 117, 46, 69, 120, 105, 115, 116, 105, 110, 103, 76, 97, 110, 100, 85, 115, 101, 79, 98, 106, 101, 99, 116, 47, 49, 48, 55, 57, 52, 57, 50, 57, 55, 47, 77, 70, 65, 45, 50, 48, 50, 51, 7, 0, 3, 0, 0, 0, 55, 55, 49, 8, 0, 0, 0, 0, 0, 0, 0, 54, 64, 9, 0, 0, 0, 0, 0, 0, 0, 8, 64, 10, 0, 20, 0, 0, 0, 50, 48, 50, 50, 45, 49, 48, 45, 48, 50, 84, 48, 49, 58, 52, 54, 58, 52, 50, 90, 11, 0, 0, 0, 0, 0, 0, 220, 155, 64, 12, 0, 7, 0, 0, 0, 80, 79, 76, 89, 71, 79, 78, 13, 0, 0, 0, 0, 0, 15, 0, 1, 0, 0, 0, 74 ] } ```