Create a json schema for the model

[fretchen]

This is a Work In Progress to establish a basic json schema reference for the model and hence to fix #76 . At the time of writing the draft it does:

• Implement a first schema.
• Derive automatic documentation of the schema through jsonschema2md
• Implement a first validation example data.
• Show that the main example template for excel is currently not compliant with the schema.

Some obvious open issues arise from the documentation, which is quite poor for the moment.

• We need to make clear that the schema is just a reference and users really do not care about this part. This is for technical people only.
• We need to make clear for users that they should a look into the Excel.
• We need to make clear that there are different kind of json files. json files with data and the json files with the schema.
• We most likely need to have some documentation of the relationship between the schema and the excel (other data formats). This is likely to be best similiar to the documentation of the location types.

I also think that we need to fix #77 before we can consider this one.

[fretchen]

This is now the first full pull request, which esablishes a machine-readable reference for the project location model. It should therefore fix #76. It contains:

• A json schema within the reference folder. The schema file is called references/project_location_schema.json.
• The automatically derived documentation through jsonschema2md. The resulting documentation file is contained in docs/dev/project_location_schema.md. Its content should be the same as within the table of the technical note.
• Implement a first validation example data for excel, csv and json all based on the same reference.
• Added new file that documents the need for a machine readable reference and the relationship between the template and the schema itself.

To document the increasing number of technical tools, I also had to do some work on the structure of the documentation. You can see the updated documentation in my private repo.

• I moved a number of documents on tors into a common docs/tor folder and made sure that the links still work.
• I made it possible to have index pages for sections.

I am looking forward to your comments. But if we succeed in merging this one, I am fairly optimistic that it could enable us to have a variety of converters and validators with fairly minimal effort.

[goergen95]

Just a few comments that came to mind by quickly skimming through the JSON Schema:

• budget share

  • should this better be bound e.g. 0-1 or 0-100?

• dac5PurposeCode

  • this should be an enum with the DAC codes so if a non-existing DAC code is entered validation fails

• geographicExactness

  • the values of the enums are not machine friendly, e.g. I would not use whitespaces and brackets here

• kcThemeSubSector

  • this also should be an enum with the sub-sectors so validation fails if a non-valid sector is entered

• locationActivityStatus

  • enums are not machine friendly, e.g. I would use only lowercase and no slashes

• locationTypeName

  • this should be an enum with a dependency on the value of kcThemeSubSector

• uniqueId

  • will this be a number or maybe a GUID?

[fretchen]

Let me go through the different points step-by-step

1. budgetShare

This must be a value in € according to the technical notes. So it cannot be bounded from above, but I added a bound from below, which I tested.

2. kcThemeSubSector, dac5PurposeCode

I see your point, but I am a tiny bit hesitent to add all the enums. These are a lot of enums, making the schema (and hence its documentation) quite lengthy. I am wondering a bit if it would not be better to put those things into code that would be part of an validator. I somehow expect that we will have to write excel and csv validators anyways and we have to handle the translations etc. So I am a bit torn between the options here. Might be a point to be solved before a release 1.1, but ok to separate out from this PR ?

3. geographicExactnessand locationActivityStatus

This is well noted, but even a larger problem as we will have even more labels from different translations. So I guess that we will have to require those values to part of a code list and only show the "labels" in the different templates? This would go in a similiar direction as we started for the locationType where each type has several attributes (including translations). Might be another point to be solved before a release 1.1, but ok to separate out from this PR ?

4. locationTypeName

I am fairly sure that the suggested validation is beyond the capabilities of jsonschema and will require a specific validator behind. I am completely sure that I do not know how to implement this with jsonschema alone.

5. uniqueId

Good question. I would guess that this an open question for the moment. @Jo-Schie ?

Taken together, I feels that jsonschema is a first step, but that it will certainly have to be integrated into custom validators...

[goergen95]

Thanks!

I guess I just got confused by the name budgetShare. To me this sounds like a percentage. Maybe one could think of a better name if it is not really a share.

There are different ways to compose complex schema, one of which is the $ref keyword with which you can compose the main schema from different sub-schemas. This has the benefit that each individual schema remains understandable and maintainable.

I am indifferent wether or not to include this (and also possible issues arising from translation) in the current PR. But handling the complexity of the specification in separate implementations of several validators is a very bad idea as it introduces a very large surface of failure for something JSON Schema is specifically designed to solve.

[fretchen]

Sounds good to me. Depending on the input of @Jo-Schie I would tend to separate the issue with the translations and the complex schemes. I agree with the schemes and the $ref as you suggested. It would however possibly involve some automatic creation from the official references. So would have to see how this would work out best...

[Jo-Schie]

i merged this for now. I think it is good to have a starting point for dicussions. Thansk @fretchen !!