initial attempt at varda converter

cholmes commented 4 months ago

So this converts the bare minimum for fiboa - id and geometry. It does work against both direct output from the API as well as from ungzipped downloads, which are geojsonl - but geopandas handles it well, so that's nice.

What I couldn't figure out with my meager python / pandas skills was how to extract nested objects. Varda uses a lot of these, see the attached file:

fieldBoundaries.json

           "properties": {
                "centroid": [
                    -1.0953325496926045,
                    45.86800874755247
                ],
                "representative_point": [
                    -1.094904370122088,
                    45.8677203482537
                ],
                "area": {
                    "value": 303981.362269,
                    "unit": "m2"
                },
                "perimeter": {
                    "value": 2499.2999,
                    "unit": "m"
                },
                "country_iso_codes": [
                    "FRA"
                ],
                "field_relationships": [
                    {
                        "type": "delineates",
                        "field_id": "WQL.XDBC",
                        "effective_from": "2022-11-10T16:16:34.449571",
                        "effective_to": "9999-12-31T00:00:00"
                    }
                ],
                "boundary_references": [
                    {
                        "varda:user_provided_key": "14246221",
                        "varda:boundary_id": "a28bc62b-a755-4118-ad03-9de3e3b5389c",
                        "varda:source_name": "FRA-LPIS",
                        "varda:delineation_type": "GOVERNMENTAL-STANDARD-BOUNDARY",
                        "varda:view_policy": [],
                        "id": "8c297fdf-bed6-4862-a5ad-aff03b8b0970"
                    }
                ]
            }

I'd like to pull up area, perimeter and effective_from at the minimum, since we have definitions of those. I think highlighting field_id and having an extension for it could make sense. For the rest I'm curious on thoughts on what to do - I think we could represent them faithfully in GeoParquet with nested columns, no? Or we could flatten them some for more geospatial tool interoperability.

Centroids, iso_code and representative_point do seem like general concepts we could just put in a generic extension.

Oh yeah, I also wasn't sure if perimeter and area will always be meters - so we could perhaps have the migration check to make sure it's meters or try to adapt if it's other values there.

@m-mohr - if you can do a couple examples of how to handle nested stuff I can likely take it back over. Or you could run with it. For the full dataset downloads I think you can just sign up for an account (maybe it needs approval? But I'm sure you can get that), and then get them from https://fieldid.varda.ag/hub/downloads

One potential very minor improvement is to allow the -i to take a .gz file and unzip. The download for varda are geojsonl's that are gzipped. So it could be nice to directly read what people will download, instead of making them figure out unzipping. But very minor request - easy to just instruct people to unzip it.

m-mohr commented 4 months ago

@cholmes

What I couldn't figure out with my meager python / pandas skills was how to extract nested objects.

I implemented dot notation access for nested objects in GeoJSON. Arrays won't work though. But this gives us access to area and perimeter at least. Updated the template accordingly.

I'd like to pull up area, perimeter and effective_from at the minimum, since we have definitions of those. [...] if you can do a couple examples of how to handle nested stuff I can likely take it back over.

I pulled area and perimeter. Didn't for the dates yet adue to the status of https://github.com/fiboa/timestamps-extension - still received no feedback. I guess I should send this out again to Bayer and others?!

For the rest I'm curious on thoughts on what to do - I think we could represent them faithfully in GeoParquet with nested columns, no? Or we could flatten them some for more geospatial tool interoperability.

As these are arrays which can have unlimited entries, it gets difficult. Maybe ignore them for now, add a todo and implement them later?

Centroids, iso_code and representative_point do seem like general concepts we could just put in a generic extension.

Centroids are probably pretty generic, we should nevertheless define what it means (i.e. how to compute them).
ISO Codes are also pretty generic, but an array here. So we need to discuss first whether we allow just one or multiple codes.
representative_point is only really useful if the underlying algorithm is the same, otherwise it may just be the centroid for many

Oh yeah, I also wasn't sure if perimeter and area will always be meters

I did check the API docs and it seems perimeter is always m and area is always m². I kept the converter flexible so that we can add more if needed. Please note that area in fiboa is in hectares!

One potential very minor improvement is to allow the -i to take a .gz file and unzip.

That should already work. Did you try something like -i downloaded_file.gz|file_in_archive.json?

cholmes commented 3 months ago

Sorry for the slow response - just getting back to this now.

I implemented dot notation access for nested objects in GeoJSON. Arrays won't work though. But this gives us access to area and perimeter at least. Updated the template accordingly.

Ah, ok - great.

Didn't for the dates yet adue to the status of https://github.com/fiboa/timestamps-extension - still received no feedback. I guess I should send this out again to Bayer and others?!

Yeah, that'd be good to do.

As these are arrays which can have unlimited entries, it gets difficult. Maybe ignore them for now, add a todo and implement them later?

Sounds good.

That should already work. Did you try something like -i downloaded_file.gz|file_in_archive.json?

Ah, cool! I didn't know that was an option.

m-mohr commented 3 months ago

What's the status of this, @cholmes? Shall we merge for now?

fiboa / cli

initial attempt at varda converter #82