Json Type Provider from Json-schema

[jruizaranguren]

Json-Schema could provide us with a principled and standardized way to understand json data. Although it has not got much traction yet is a powerfull and complete standard with a couple of decent implementations.

It migth simplify type inference and reduce ambiguity. It also allows for shareable lightweight data specifications.

A brief brainstorming of json-schema keywords that could be used to leverage F# characteristics:

• "additionalProperties": differenciate record type vs classes.
• "oneOf": specify discriminated unions (also with "enum")
• "required": for "option"
• "maxItems/minItems": array cardinality (validation)
• "uniqueItems": set behavior in arrays.
• "multipleOf,maximum,minimum": for selection of numeric types with desirable precision.
• "maxLength,minLength,pattern": for specific string validations.

We would also have a standard way to:

• nest and reference schemas through URIs.
• richer semantics for validation of data
• types could be named (even a namespace convention could be generated from schemas).
• room for creating more performant parsers dinamically
• specify APIs, as in heroku and google discovery service

I'm willing to contribute to this approach, I have worked with json-schema a couple of months.

[ovatsus]

Sounds great! Send in the PR :)

[jruizaranguren]

PR?

[ovatsus]

Sorry, Pull Request

[tpetricek]

This would be a nice feature to add - and it is actually not that hard to implement!

The key thing is to parse the JSON schema and, based on the schema, generate value of the InferedType discriminated union (you can see It here), which is our internal representation of a type. Once we have that, the rest of the type provider can generate a nice .NET type from the internal representation (including things like options).

Aside from that, you'd have to add static parameter (say Schema) to the list of parameters, and change the code that reads the sample to instead look at the schema (and produce InferedType)

[jruizaranguren]

Thanks Thomas. I will have a look at the weekend and see if I can make a first draft. Could you give an estimation of the effort needed for this feature? (I will multiply for 10 to get my own estimate anyway :-)

[tpetricek]

It is difficult to give any estimates, but I believe that this is one of the easier (but fairly significant) features to implement! It is mainly a matter of parsing JSON schema (possibly using our JsonValue that is already in F# Data) and extracting the information from there into our internal structures. But I have learned enough about software engineering to avoid giving any estimates :-).

Also, when you start working on this in your repository, feel free to send a pull-request with the work in progress, so that we can give you more feedback (and answer any questions you may have).

[blumu]

Jet.com has released sources for a related project: https://github.com/jet/JsonSchemaProvider

[jruizaranguren]

@blumu, it depends on Json.Net and it just conforms to JSON-Schema v3 (deprecated). Newtonsoft released a JsonSchema library, but it has license restrictions.

[tony-johnson-i2o]

Was work ever started on this? Are you still looking for someone to pick it up?

[realvictorprm]

I wonder too

[realvictorprm]

@tpetricek can we get this going again? Is it technical seen still relevant?

[tpetricek]

@realvictorprm I do not know how widely is JSON Schema adopted, but aside from that, this still sounds interesting and relevant, so if anyone wants to have a go, I'm all for it :)

[realvictorprm]

Good, need this for glTF which specification is in schema json. Can you point me a bit?

[realvictorprm]

@tpetricek I have some questions which need to be answered so I can start developing a new up to date json schema type provider:

1. Do we use an external library for parsing the schemas or should I write a complete new one? 1.1 If I should write a new one, on which existing code should I write the parser. Do we have helper functions or restrictions on how to write the parser? (I would go with parsing the schema as regular json file and afterwards visit each field to validate and create a correct schema type)
2. Must it be an erased type provider then?
3. Is it fine to go with the latest json schema version?
4. Anything else you might want to tell me? (e.g common mistakes in contributing to this repository)
5. Should I amend my commits everytime I make small changes or is it fine to make commits for reverting and re-factoring an algorithm?

[realvictorprm]

@tpetricek feel free to assign this to me.

Tracking PR is #1069

[DoubleCouponDay]

I'm confused as to the original posters intent for a pull request. Do you wish to allow json schemas to become a JsonProvider type for multiple json files, which implement the same schema? Currently I don't think this is possible since JsonProvider still includes json-schema specific keywords, such as Properties.

The best workaround I use is to use any one of the schema implementing files as the JsonProvider's source.

[blumu]

@doublecouponday that's right. I meant a type provider generating types based off a JSON schema file. The suggested workaround is not meeting the requirement, in particular a single sample JSON cannot always represent all the possible JSON files accepted by a given schema.

[olstakh]

This would be so great to have! I have a list of json files that all follow the same schema. I would be great to load JsonProvider with a schema and then parse individual files! Without it, i have to have a provider for each separate file, which is not scalable

[kurt-mueller-osumc]

This would be a nice feature to add - and it is actually not that hard to implement!

The key thing is to parse the JSON schema and, based on the schema, generate value of the InferedType discriminated union (you can see It here), which is our internal representation of a type. Once we have that, the rest of the type provider can generate a nice .NET type from the internal representation (including things like options).

Aside from that, you'd have to add static parameter (say Schema) to the list of parameters, and change the code that reads the sample to instead look at the schema (and produce InferedType)

Say I wanted to give I wanted to pick this up and start my own attempt a pull request for this feature. Any advice from @tpetricek or @dsyme as to where to start? I was going to look at how the XML Type Provider uses XSDs to generate types.

[kurt-mueller-osumc]

Jet.com has released sources for a related project: https://github.com/jet/JsonSchemaProvider

Does anybody have a copy of this repo? It no longer exists.

[xperiandri]

There is a newer one https://github.com/florenzen/JsonSchemaProvider It works but it is not production-ready. The author agrees to reuse that code to FSharp.Data. I tried improving it and migrating it from NJsonSchema to JsonSchema.Net here