json-schema transformation and normalization specification

[nickl-]

While considering the plausibility of and use cases where one might require different value types for a single type definition https://github.com/json-schema/json-schema/issues/38#issuecomment-11461515 an alternative approach came to mind, data normalization through transformation.

given the following json data example

{
    "animation": "on",
    "rotate": "off",
    "show-splash": true,
    "hide-desktop": false,
    "repeat": 1,
    "delay": 0,
    "screensaver": "enabled",
    "password-protect": "disabled"
}

While validating (more specifically while transforming but since validation can only occur once transformation is complete there is a dependency) fields defined with simple transform strings can process the data and produce an altered form.

{
    "type": "boolean",
    "transform": ["/on|enabled|1|true/true/", "/off|disabled|0|false/false/"]
}

Resulting normalized form after validation.

{
    "animation": true,
    "rotate": false,
    "show-splash": true,
    "hide-desktop": false,
    "repeat": true,
    "delay": false,
    "screensaver": true,
    "password-protect": false
}

As a separate spec which can be presented to IETF as additions to the json-schema specification we should consider:

• the different use cases where this might be useful/applicable
• research and compare notes with any current practices of transformation (XSLT comes to mind)
• consider the different approaches by which to facilitate the requirements
• reach a consensus on the target requirements and optimum implementation
• draft RFC document
• write reference implementation for transformation
• submit completed specification.
• etc.

[geraintluff]

Some kind of JSON Transform standard could be cool, but it's so far from the concerns of JSON Schema, I'm not really sure it should be an "issue" on this particular repository.

So I think that being an "addition to the json-schema specification" is probably not the best way to phrase it - it's a completely separate standard, just one that might reference JSON Schema.

But yeah - it would be interesting to do.

[fge]

There is already an IETF draft for JSON Patch (although the syntax is quite different), doesn't that fulfill this kind of needs already?

[geraintluff]

JSON Patch is like a diff - it's tailored to a specific document. What Nick was talking about was something closer to what RegEx search/replace does for strings, or XSLT does for XML.

So, say you're taking JSON data from a bunch of different sources (you're aggregating their data), and presenting it in your own unified format. The idea is that instead of writing a translator in code, you could write a JSON Transform document that specified the changes that needed to be made.

It is, however, outside of the core interests of JSON Schema, so I'm wondering whether an issue on this GitHub repo is the right place. Perhaps a thread in the Google Group? Discussions there seem a bit more wide-ranging there.

[nickl-]

It is, however, outside of the core interests of JSON Schema, but it's so far from the concerns of JSON Schema,

I was not aware of any specification that defined this so called core interests or concerns, care to elaborate? As an inanimate object it should really not be interested in any concerns if you ask me.

I'm not really sure it should be an "issue" on this particular repository.

Nor was aware of any constraints as to what is considered acceptable or not as an issue on this or any other repository. Care to elaborate? Are there any other sensitive topics I should try to avoid as to not overstep this boundary again?

@geraintluff What exactly offends you so about the mere proposition that you'd rather have it banished than to hear it out for what its worth. At least ask a question, you replied twice but there is no room for responding you decided this is invalid end of discussion. even the title is disregarded as wrongly phrased with no suggestion presented as alternatives or examples. Nothing to continue a conversation on.

it's a completely separate standard, just one that might reference JSON Schema.

What exactly is that supposed to mean? Maybe an example of how this completely different standard might look (is it like YAML or am I still cold?) and what exactly referencing JSON Schema would/should/could produce for this standard to perform such a task? Just the sheer fact that it should not even be mentioned in this forum makes me wonder how it would have any knowledge of JSON Schema or what it would entail to reference it.

json-schema validation does not produce anything other than a schema document nor does json-schema hypermedia and link produce anything JSON Hypermedia documents but JSON Trarnsform document is what is required here. I wish I had the faintest idea what such a document looks like? What does it produce after it managed the "referencing" with JSON Schema? How does it know where to get the Schema in the first place since it is completely separate to the whole concept or any concerns of JSON Schema? If I had something to work with and didn't have all these questions? If you had a question at least then I might've had something to explain to you but I'm completely drowning here... Who are you even talking to it seems like you're addressing some judge or moderator and you were called away from a busy day to give your opinion to the board.

You seem to understand the use case, as I con tell from your explanation of JSON Patch. But I never said a Transformation document did I? Maybe you didn't understand my schema:

{
    "type": "boolean",
    "transform": ["/on|enabled|1|true/true/", "/off|disabled|0|false/false/"]
}

How about this? Is this JSON Schema or also subject to the same omission criteria that got the first post booted off the premises?

{
    "Title": "Desktop configuration settings",
    "description": "Configuration settings for switching features of y funky desktop on or off.",
    "type": "object",
    "definitions": {
        "configSetting": { 
            "type": "boolean",
            "title": "Configuration Setting",
            "descriptin": "Allowed values for tranformable configuration settings are on,enabled,1,true,off,disabled,0,false",
            "default": false,
            "transform": ["/on|enabled|1|true/true/", "/off|disabled|0|false/false/"]
        }
    },
    "properties": {
        "animation": { "$ref": "#/definitions/configSetting" },
        "rotate": { "$ref": "#/definitions/configSetting" },
        "show-splash": { "$ref": "#/definitions/configSetting" },
        "hide-desktop": { "$ref": "#/definitions/configSetting" },
        "repeat": { "$ref": "#/definitions/configSetting" },
        "delay": { "$ref": "#/definitions/configSetting" },
        "screensaver": { "$ref": "#/definitions/configSetting" },
        "password-protect": { "$ref": "#/definitions/configSetting" }
    }   
}

What exactly is so strange about the concept? Oh wait I am assuming I have JSON Schema here scratch that

What exactly is not json schema about that? Not hardly as strange as format which does no formatting and patternPropperties? Really? you're going to swallow that which is on the verge of breaking valid JSON but transform we can't even talk about?

I wish I know what you are seeing cause tho most far fetched that I can think of would be some AWK type transform but even that would not cause me to say WTF? Hypermedia is using URI Templates that is fine transform is using pattern which is already json schema what am I missing.

The next spec: 'json-schema aho weinberger kernighan interpretation"

{
     "format": "pipe",
      "type": "stream",
      "awk": [ { "BEGIN": "x=5" }, { "/pattern/": "if(NF>5)x+=$6" }, { "END": "print x" } ]
}

Don't worry I will start a new repo for my "JSON extensions to basic primitives: Vol XIV Stream wrappers and STDIO PIPEs" I won't subject you to that mind blowing screams but that awk keyword looks prefectly json-schema'ish to me we can even tap into the routines fo patternPropperties to first identify the pattern before we eval the expression. coming to think of it... haw about json-schema CLI, readline and terminfo - clipping the wings of unicorns as a first tier security response to invalid concerns and interests. hmmm maybe that is too long ok I agree lets drop the ANDs

Throw me a bone here already... =)

[geraintluff]

@nickl-

I didn't mean to offend you. What I meant was that I didn't immediately see how the discussion would lead to changed wording or additional keywords in the JSON Schema standard.

it's a completely separate standard, just one that might reference JSON Schema.

What exactly is that supposed to mean?

JSON Schema describes the form of JSON data, but it is passive - it "describes" data, instead of being "applied to" data. Validating against a JSON Schema multiple times will result in the same outcome. However, applying a JSON Transform might not (depending on the nature of the transform).

However, JSON Transforms (or whatever you call it) would be an extremely interesting line of development, and one I would like to see happening. I can totally see how JSON Schema would be useful when defining a JSON Transform language. For example, the opening of the spec might say something like this:

This document defines the JSON Transform language, which harnesses the pattern-matching capabilities of JSON Schema to define rules for systematically altering data.

I think a transform language based on JSON Schema would be awesome. If a project were started on it in a new repo, I would love to join in the discussion.

[nickl-]

@geraintluff

I didn't mean to offend you.

No offends taken... if I came across as being annoyed it was due to the fact that I had to grasp at straws to try and reply to you nonchalant response to the topic. Thank you for placing some motivations around your statements I can understand now where you are coming from and we have a platform from which to continue discussion.

off topic: This is not personal b.t.w. and if taken personally there is going to be loads of heartache and pity along the road. This is purely for the greater good of everybody and we are only the mechanisms facilitating a change, creating order amongst chaos to arrive at a mutually beneficial and acceptable conclusion. If we accept that comments, suggestions, disagreements, etc are directed at the content not the person who actually placed the words after each other, only then will we be able to see this through to the end. There is no room to get offended or insulted it is not yours to feel pride for. We are charged with the duty to speak the peoples voice and reflect that in our writing. If anyone disagrees we need to change that or convince them, by pointing at the content or choosing to word it differently, so that they can agree as the majority does that what is said in any case, is what they wanted after all.

JSON Schema describes the form of JSON data, but it is passive - it "describes" data, instead of being "applied to" data.

Actually:

a JSON based format for defining the structure of JSON data. JSON Schema provides a contract for what JSON data is required for a given application and how to interact with it.

I don't see anything about it having to be passive.

Transformation is a definition for the data allowing for different ways to interact with it by letting you assign different valid values for the property/field.

Transformation still extends the contract of what data is allowed. The definition dictates what the result will be so there are no confusion or uncertainty nor should anything be unclear. This will not be the case if you are applying an external construct to the data, as you are suggesting, the meaning will be lost then.

Transformation allows for normalization of data, Names in Title Case or identifiers in ALL CAPS the meaning stays the same and the end result is expected because the data definition defines it as so.

If the transformation schema is applied you can expect normalized data after processing the transform. You do not have to use the transformation schema in your definitions, you do not have to transform the data. If you choose not to then the flexibility it provides(can provide) can be disregarded and is lost to you because the data will now fail validation. So what are you afraid of? For the rest of us that have no qualms and a strong need to have this functionality, if it is all right with you, can we go ahead and use it?... do you mind?

Lets maybe look at it from the other end as well in hope to cover all aspects once and for all:

What are the benefits to applying an external construct that is not part of the data definition? What do you gain by the fact that the data definition is passive and the knowledge that nothing can be changed? Why would you want to impose limitations on what JSON Schema can and cannot do is this based on a lack of vision, a fear of change or a fondness of something that was so anything else must never be? Keep in mind JSON Schema doesn't exist, never has and we are in the process still of defining it. The drafts have all expired they mean nothing, they are obsolete and we are deciding here and now what it might be for generations to come. What on earth is the purpose of splitting the community by creating a new repository for the purposes of accomplishing the same goals? Is there too much activity here that you think we can't handle some more? It is called JSON Schema transformation for the purpose of presenting it to the JSON Schema community, who do you want to address in this separate repository? What are the benefits?

You do not have to agree with everything JSON Schema, you do not have to adopt everything JSON Schema. When we decided to create smaller concise documents it was for the purpose of specifying separate and loose standing additions to the whole. If this is to be the non passive side of JSON Schema then it is still JSON Schema, still uses the same artifacts and still applies to JSON data. It therefor stays part and purpose of this same community with interest and motivations to see these specifications come to light.

You already agree there is benefit. You already agree it looks like JSON Schema. You are ready even to get involved with its creation. Why do you not want to tap into the resources we have so painstakingly brought together for this sole purpose. What else is it if not JSON Schema? Maybe then I can understand why we are wasting time on deliberating whether it is or is not where we could've already been discussing what it is, what we want from it, how we can apply it and get it into document. What are you trying to accomplish here? Will you only see these 3 documents through and then abandon this place as work is done? Again what are the benefits as I can see none.

Now please do something constructive instead of just being a obstacle in the road. If you do not see transformation as the examples I have brought to the table then kindly present what you see this external non JSON Schema alternative will look like because you have not yet convinced me that it is even possible let alone plausible, functional, practical or based on any motivation preferable to what I have in mind. The floor is yours...

[geraintluff]

OK, I'll have another go at saying this:

The goal of JSON Schema is to describe the data - that's what "schema" means. A schema describes what shape the data should be, but it does not define how to change it.

In contrast, a transformation language can specify how to change the data. They are very separate goals. In the XML world, XSLT is separate from any of the XML schema standards for this reason.

So: what I was suggesting is that a separate project could be started for a JSON Transform specification. It could reference/extend the JSON Schema standard, but it would not be part of it.

What else is it if not JSON Schema?

A schema describes the nature or the shape of the data. If you're changing the data, it is not a schema, it is a transform.

[fge]

I agree with @geraintluff here, no need for a transformation specification in any sort in JSON Schema.

[nickl-]

I agree with @geraintluff here, no need for a transformation specification in any sort in JSON Schema.

And that concludes that, transformation of any sort is out of the question. The master has spoken and concludes without any further discussion.

Now what is the harm I wonder if someone were to write a transformation spec for JSON Schema, someone else mind you as as we could've suggested to @P-Product to -start a Templating spec but that too is shunned. What is the harm I wonder was this not the original goal for JSON Schema, why yes it is #4 what have you.

@fge JSON Schema has several roles: validation, documentation, hyperlinking (maybe more?).

Obviously Transformation and Templating do not fall under those "maybe more" categories. Even the mere mention of it is not constructive I wonder why that is? Is it instead constructive to completely shun an idea? Now that will be something new to me.

[alexweissman]

I know this is an old (and it seems contentious) thread, but has there been any renewed effort to develop a JSON Transform standard?

I am very interested in using JSON Schema along with some standard for specifying data transformations in my projects (UserFrosting and Bootsole)

Standards projects like this are extremely important, I wish there were more!