Create a json schema for the model

[fretchen]

We now have the specifications for the template fixed within the technical notes. To make progress on this we could use these notes and translate them into a json schema. The advantages of json schema to the markdown format:

• It is directly machine readable.
• It can be used as an input for the generation of markdown files.
• It as a very broad support across different technological tools
• It can be used to directly create web forms.
• It plays nicely with python / pydantic, which is under the hood of geonode.
• It is designed for complex data formats. Hence it could allow us to provide a technical bridge to more advanced systems as suggested in #10, #18 and #21.
• Any technical implementation could then be automatically tested if it is compatible with the jsonschema, lending to much more flexibililty and robustness.

Alternatives

I currently do not see any good alternatives. Possible options would be:

xlsx

• Hard to read for machines.
• Hard to implement complex data structures.
• Not really a very open standard.
• Not very broadly used as reference.
• more adapt as a technical implementation than as a reference

markdown

not machine readable and hence not able to serve as technical basis for validations.

uml

• this is a fairly abstract format which does not allow for validation.
• so it falls more into the documentation level and allows for less precise implementations.

direct technical implementations

Keeping them compatible requires a common reference / language. This should be digestable across technical implementations. Hence, jsonschema.

I would propose this as first step to see where we can go with this. Comments @Jo-Schie , @Maja4Dev or @goergen95 before I start simple first attempts in this direction ?

Originally posted by @fretchen in https://github.com/openkfw/open-geodata-model/issues/24#issuecomment-2187096958

[goergen95]

Great to see those first steps on this! This looks really promising. Some food-for-thought to keep in mind when working on this:

• as a human, I find JSON hard to write and most of the time I make a lot of mistakes with the brackets etc. when I am trying. Since we should aim for everyone to easily participate in the maintenance of the specification, we could think about moving it to YAML and create JSON Schema automatically from there?
• sooner or later we will have to decide if we want to include the notion of a location collection in the specification, that is a collection of 1 or more project locations. We could then validate a collection as a whole instead of its single features, and it would enable us to think about variables on the collection level
• eventually we will have to think about the representation of the geometries. This current approach is very far from ideal and I wonder if the geometry should be part of the specification. I tend to yes. From my perspective, this would mean for an potential excel converter to create a GeoJSON collection with all geometries included which can then be validated

What are your general plans on proceeding with this? I would really like to see a conversion (Excel -> (Geo)JSON) and validation tool which we could test against real data, what you think?

[fretchen]

Let me try to puzzle through the topics.

YAML versus json schema

I completely see your argument and would say that this be cool to have. Even more so as I expect constant confusion about json files that are schemas and json files that contain data. However, as for the technical implementation, I think that there are a few things to consider. As those points belong directly into the discussion #24 I will those points over there.

Collections of locations

I must admit that I am not sure that I understand the challenge here. In my naive view I would say that each excel file would already be a collection of rows and hence locations. What am I missing ? As this sounds like an extension of the model and a "new" feature should we spin up an issue for this ?

Geometries

Again, I must admit that I am not sure that I understand the challenge here. We currently have the possibility to add longitude and latitude. And we can specify if we have point geometries or not. Could you provide an example of what you would find better and what you would like to have.

Data conversion

As you might see I have started to validate excels and json data within #79. Once this validations work, I think that a converter would be quite easily feasible. Most crucial would be most likely converters between xlsx, csv, kml and json. However, I suppose that work on this can only really start once we have a machine readable reference (i.e. if #79 is accepted).

[goergen95]

YAML versus JSON schema

please see my comment in #24

collections of locations

My comment is about at what level we target the specification. Does it specify single entries and validation is done in a loop with no notion of a collection or does it somehow include that notion (e.g. multiple entries that share common information such as "project number").

geometries

In Excel, you can only reliably collect point locations. More complex geometries produce a mess, in my POV, however, the specification does not seem to target only point locations (at least, that is not what I experience in practice). How will this be part of the specification, how to convert reliably and how to validate those geometries are all open question which need to be taken care of eventually.