Anthropic commented 8 years ago

EDIT: This is now being tracked at json-schema-org/json-schema-vocabularies#2 as its own project.

Please join us there for further discussion!

I propose a separate schema extension for the view definition in the same way hypermedia was separated into its own schema.

JSON Schema Form has a form definition that is array based for ordering, with the majority of its attributes taken from json-schema. It also has titleMap array to define labels for enum fields and can define field types (including custom field types) to use for displaying the field.

As it describes nothing more than a type name it is generator agnostic and not destined to be specific to our Angular implementation. In fact we are already working on making a plain JavaScript version and there is already a React implementation.

The main purpose of raising this here is to highlight a need, in our view, to avoid allowing view based attributes to diminish the data definition purity of the json-schema specification.

Choices and Enum Names would both bring changes to json-schema that would confuse any existing view based frameworks like ours and others while bringing little value and breaking model view separation of concerns. A full json-ui-schema would provide the missing view properties while keeping view and model separate, as they should be in our view.

In addition to a ui-schema, the same way the main json-schema has an extension for hyper media, we would also like consideration for a json-rules-schema for controller/field logic (if/when etc..) and json-flow-schema for workflow/wizard/state management keywords.

I am interested in feedback primarily on the json-ui-schema proposal. But all feedback is welcomed.

Proposal: JSON Schema UI vocabulary Child issues:

252 2017-02-16

erosb commented 8 years ago

I welcome this proposal. It is a much better idea than trying to hack UI-specific keywords into JSON Schema.

handrews commented 8 years ago

I'm definitely in favor of a UI-oriented schema extension. I have no idea whether it is reasonable to include it directly in this core project or whether it should be an independent effort. Either way, it would be very helpful to be able to respond to UI-oriented proposals with "please make this proposal in the UI extension".

We would need to come up with some way of determining what annotation properties, if any, would stay in the existing three parts of JSON Schema. In my opinion, anything that is useful for documentation (like description) or instance processing (like default) should stay in the validation document.

On the other hand, enumeration names should probably go in a UI extension because description is sufficient for human-oriented documentation. The individual enum names are only needed for matching them to values in the UI.

handrews commented 8 years ago

@Anthropic if you would like discussion on the json-rules-schema idea, could you please file it as a separate issue? I think it is too different from json-ui-schema to discuss in the same issue without the whole thing becoming confusingly muddled.

Anthropic commented 8 years ago

@handrews the other extensions I consider as something to discuss after we determine how to approach json-ui-schema. I already created an org of that name just in case.

Regarding properties, I see description as being used more for what you would write in a MySQL field description rather than as a field hint like some use it now. I would like a separate hint field for that.

With json-schema-form we currently allow most json-schema fields to be overwritten in the ui-schema as it is very often the case that an online form will only display part of a data definition. As such we do not validate for fields not added to the ui-schema or removed by logic rules. Obviously a field may still be hidden. You can see our current behaviour at http://github.com/json-schema-form/angular-schema-form documentation.

If interested, I can created a repo in the org I registered and add all json-schema admin interested as admin to that also? Then create a rough draft spec based on our usage as a starting point. Also worth considering to keeping it within this org too, probably easier if it is, just means giving me admin rights on a json-ui-schema repo here to get started.

Anthropic commented 8 years ago

@awwright I finally made the issue for what I discussed with you previously

awwright commented 8 years ago

@Anthropic I've been following this, but you and everyone else seems to have already said pretty much everything I had to say, haha.

I'm very interested in building UI's from a JSON Schema, so I'd like to advance whatever would help accomplish this.

What you describe follows my experience with building user interfaces from JSON Schema pretty well, especially that even for a single JSON document, you'll want to be able to create multiple different forms to edit it in different ways. For example, editing your own comment on a blog, vs. moderating comments on a blog.

There probably should be some way to specify a "default" style, though. If I'm browsing a JSON API annotated with JSON Schema, my user-agent should still be able to show a sensible rendering of an editable JSON document.

Anthropic commented 8 years ago

@awwright well the way that angular-schema-form handles it, we allow users to define an x-schema-form extension object type property on the schema for users who for whatever reason cannot work with two files. But by making it a single property it still clearly maintains an identifiable separation of concerns. We do only recommend it for very small forms though.

If we rename and allow perhaps a ui-schema property to do that, it would still make it part of the ui-schema definition extending the main json-schema spec. Also a schema could always be tied to its ui-schema by a top level ref... or even an array of oneOf ui-schema ref...

awwright commented 8 years ago

JSON Schema Core defines a framework to allow an arbitrary JSON document to be described in any way. I imagine a UI vocabulary would just be another vocabulary following the JSON Schema paradigms, and application/schema+json media type.

Suppose we had entirely different vocabularies for validation and UI. If you want to describe a single document with both vocabularies, you could set two rel=profile links on the instance (one per vocabulary). Or just use one, and do something like:

{
    $schema: "http://json-schema.org/draft-04/schema#",
    type: "object",
    /* ... */
    allOf: [ {
        $schema: "http://json-schema.org/example/json-ui",
        fields: /* ... */,
        /* ... */
    } ]
}

Anthropic commented 8 years ago

@awwright our ui-schema includes all the same values in the main schema with type defining templates to use (so a enum type in the schema can be type select in the ui-schema).

The main differences to core schema in our implementation being:

for the sake of always maintaining order the definitions are within an array
it includes a key value pointing to the schema def (the array item can either be the key value or an object with key defined, it isn't currently a jref as at the time it was unclear how to work with arrays at the time)
there is a titleMap for enum labels (name, value, group and any additional properties)
some basic conditional logic processing, currently quite "Angular" but looking to change that.

There's more to it, you can see the docs here at the ASF repo

So a vocabs could work, are you thinking json-LD style? I like your example. I'm assuming that fields would be like the array in our definition, but also able to take a $ref?

Obviously I don't expect everything we do to be adopted, far from it, I know some things I don't even want to keep (like functions on the object) so I am very open to making changes! But I was thinking there's enough in our definition to get discussions started depending on where you feel is best for that to take place.

nicklasb commented 8 years ago

Hi @/all, When @Anthropic, me and some more people discussed the json-rules idea, I created the json-rules org and repo here: https://github.com/json-rules/json-rules

We used gitter, but I guess we've come far enought to discuss publicly, I have started a thread here: https://github.com/json-rules/json-rules/issues/1

awwright commented 8 years ago

@Anthropic I believe it's similar to JSON-LD; the vocabulary in use is selected by "$schema".

When I've been looking for a UI solution, there's two requirements (iirc) that I can think of, that vary slightly from what angular-schema-form is covering:

One is to assist browsing of JSON APIs that would otherwise be unstyled. So if I open up my (JSON-aware) user agent, punch in a URL of a JSON document, I can see it, I can choose to edit it it, and I can persist those changes (with a PUT request). A UI schema would be used to do things like declare data relationships ("this field points to an existing user ID") and processing ("readonly: this field is set exclusively by the server -- don't define it, or whatever it is now, don't change it, changes will be lost")

The other is that I already have an HTML form, and I want to generate a JSON document from it, and provide localized (translated) and field-specific highlights for errors.

Or maybe some hybrid. Maybe I have an HTML website, a JSON instance (e.g. representing a comment on a blog post), and I want to present it to users, editable, with some values read-only or not used at all (for example, their username, IP address, and date of submission).

It seems like angular-schema-form only handles this hybrid case right now.

We need to figure out which of these cases overlap and which are really different technologies.

The big thing is that anything going over the wire gets described by JSON Schema Validation or JSON Hyper-Schema:

Sometimes abstract resources (a blog post) will have multiple JSON representations. A blog post might have a representation aimed for the public, and a representation for moderators that includes a revision history and IP addresses.
Instances can describe how they relate to other instances: read only (set only by the server), relationship to another resource that must exist, etc.
form fields might choose to render themselves differently depending on the validation restrictions (e.g. automatically use <input type="number"/> for type:"number" properties).
If the value we want to send would be invalid, we can try to trace the invalid value back to a particular form-field and display an error there.

(Some of these features might not be well-defined yet, but they would be good fits for JSON Hyper-Schema.)

So even without a dedicated UI vocabulary, there's still a lot of information for a form generation toolkit to work off of.

What other features do we need, can we come up with a list?

Anthropic commented 8 years ago

@awwright do you want to pick a spot in the wiki for planning to get a start on a requirement/use case list? It could get rather difficult to track if we start posting requirements and use cases in numerous posts on this issue.

I totally agree that any data transmission should be defined by those two schema, that's why when I implemented remote enum option retrieval I used the hyper-media link with a rel of 'options'.

At the same time there is obviously, based on other issues raised, a consideration to re-evaluate the hypermedia schema which could go hand-in-hand with any requirements garnered from a considered requirements list for a ui-schema.

mmc41 commented 8 years ago

Good initiative. I am also interested in this. I am however, a bit unsure about how json schema extensibility works (what is possible / what is not) when using a schema extension for defining the view. F.x. Can existing json schema elements be extended in this way or do we have to create new, more powerful alternatives ?

epoberezkin commented 7 years ago

@Anthropic we've been also experimenting with different form definition approaches: in https://github.com/milojs/milo-ui and in https://github.com/jasoniangreen/iso-form

I think it would also need to support grouping and, given than grouping leads to a different view structure from the data structure, the way to map UI elements to data pointers (e.g. dataPath: '/customer/name' in UI element for the first name).

Also, in general case, it's not possible to derive the requirements to a given field from JSON-Schema, so you'll either have to require to use a subset of JSON-schema, similar to what swagger is doing, or to derive JSON-schema from form definition.

I prefer the latter approach, restricting JSON schema is my main objection against swagger.

Anthropic commented 7 years ago

@epoberezkin our approach is working for the most part, we have thousands of users already using the tool and the requests for new features are fairly stable so we tend to know what the users need the most as they keep reminding us we haven't done it yet :)

The main thing we do is merge a schema with a ui-schema in processing to then generate the form. We do not validate items not found in the form definition as we generally find users do not want to validate fields if they are not shown for a selection of users for example. They can still have hidden fields to force such validations. We are migrating our core to a stand alone library which you may find of use if you take a look at it.

The current idea on this proposal after suggestion from @awwright is that we start creating a list of things that need to be managed/supported to then start defining requirements. I'm not sure how @awwright wants me to do that but hopefully he can let me know now so I can get started in a format that will be of use to him :) ps. if none of that made sense it is 1:15am, forgive me.

epoberezkin commented 7 years ago

I understand. Your approach approach is definitely useful. I just think it is not comprehensive enough to be standardised as is - it leaves many real life scenarios not covered. Flat forms, where the form structure matches data structure, while quite common, don't cover a sufficient share of use cases.

We do not validate items not found in the form definition as we generally find users do not want to validate fields if they are not shown for a selection of users for example.

I said the opposite - for a general JSON-schema it may be impossible to validate item that IS found in the form definition:

it may be dependent on other field(s)
it may be within allOf/etc - so while it is possible, it may be difficult to extract schema for a single field.

Given that your library only supports a subset of possible JSON schemas, it is useful to understand which subset it supports.

Anthropic commented 7 years ago

@epoberezkin It's ok, I don't consider it ready to standardise as-is, I only thought of it as a minimum starting point to be expanded and added to... and changed.

I think you know already but to clarify for others, when I said form definition, that is our ui-schema, currently the main feature missing is $ref support, it is possible to make anyOf/oneOf work via an add-on, but it isn't complete as part of the main lib either as we are trying to figure out how to deal with those scenario as they don't lend themselves to easy generation as they are, I'd like to see changes in that area.

As we combine the schema and ui-schema definition we have a contained validation, but we would only validate a parent oneOf/anyOf if it was included in the form definition. So you have choice over how much of the schema you want to validate against. The only issue we see coming up is if there is inclusion of $data references that would cause issues if the related value was not in the form. But at the same time you can't defend against poor implementations, in my opinion it is up to the designer to determine that they include what is required.

the-t-in-rtf commented 7 years ago

Is there a draft of the proposed extension yet?

Anthropic commented 7 years ago

@the-t-in-rtf not yet. We don't have a complete requirements list yet.

edited: changed my mind on where the rest of the comment should have been posted...

handrews commented 7 years ago

@Anthropic Is there anything here to do in the main json-schema project for this, or should work and discussion of this sort just be referred to https://github.com/json-rules/json-rules ? In which case we should close this issue here. And perhaps open an issue on the web site to link to the JSON Rules project?

Anthropic commented 7 years ago

@handrews well, given hypermedia is part of this repo then UI Schema could be here also, JSON Rules also I would expect. However there is an obvious argument for all of them to be within separate repo within this org. We initially made separate org and repo after the original json-schema repo appeared to be abandoned.

UI Schema needs to have additions such as a widget descriptive properties, input and label classes / tags, while also working with Hypermedia elements and Rule elements as it also needs to make it possible to load elements dynamically. A hint, label and title map for enum etc... that's before starting on rules where as I see it there is a further break down and potential for separation into policies for dealing with just display attributes, rules for triggering functions/scripts, rules for determining when validation can occur and so on.

So I am open to however @awwright and other contributors like yourself want to separate concerns, but I think consistency is important, so I do think if it was in another repo probably the reevaluation of Hypermedia could be done outside this repo also, thoughts?

handrews commented 7 years ago

There are a couple of proposed extensions at this point- I'm going to start a thread on the google group since I think there's a broader topic of how to arrange and manage these. Then we can come back to issues to track specific actions.

handrews commented 7 years ago

OK, here's the thread: https://groups.google.com/forum/#!topic/json-schema/cG4HAyerqQk

Anthropic commented 7 years ago

@awwright so to get to your question on the other thread that got sidetracked, the implementation we have is capable of working on different devices, for example I am working on an Angular Material display that can work on any screen size. There are already libraries that work with JSON base elements that can render on any screen. I don't see any issues with a cross-platform cross-language ui definition that can't be overcome.

Anthropic commented 7 years ago

@epoberezkin

you are leaning towards an independent format similar to what I was describing, but for some reason you are using JSON schema as part of UI definition, rather than just a validation concern.

I have never considered json-schema to be for validation alone, I have considered it a schema as the name suggests, a representation of a model. It has a validation extension and it looked like there were a lot of issues and proposals that would have made ui guidance a part of the core and I do not want to see ui needs embedded within core or validation vocabulary that would conflict with ui implementations without it being considered carefully. I do know that some libraries prefer blending ui into the schema and some do not, which is why I felt it is worth exploring and opening up a conversation on how it should fit together and what level of flexibility should or could be accommodated.

Interesting what @handrews will say, but I think this amount of data inside ui keyword seems very much stretching what a vocabulary should be.

I appreciate you taking time to come up with ideas on #252 regarding vocabulary, @awwright had previously suggested a separate vocabulary $schema root which was why I imagined a ui keyword as being such a root, but after reading @handrews' comments I may have misunderstood the idea. I will discuss it with them further to understand intentions and goals.

On the other hand, the last approach that I suggested, when you use schema for view model, rather than for data model, with UI keywords "interspersed" in schema seems to be fitting "JSON schema vocabulary". An additional consideration in this approach is that it requires defining an additional concept - a view model, and how any given UI could/should be mapped into a data structure.

So I do find that this style works for most cases, but for the needs I have I need to be able to select from multiple views of the entire model, so there needs to at least be an ability to define the ordering and parentage of the ui elements. I can imagine after seeing your suggestion a view schema not dissimilar to yours however we, and many other frameworks for ui, offer the ability to wildcard the remaining fields from your schema, that's why the other issues was created, *to get a clearer consensus on how implementers can consistently deal with a schema key of ``**. When you tried to clarify my goal your wrote 'it is impossible in general case to generate "form UI" from any given JSON schema' which I would suggest you can very much do with assumptions, you just can't do it in a consistent manner that is universally accepted as a standard.

Anyway the goal of generating a ui from json-schema will never go away and for many implementations it will remain that the ui schema is valid json-schema which means at the very least we need to be mindful of how new properties in either vocabulary will be interpreted, I recall you said to Henry somewhere that the hyper-schema needs to consider the complexity of validating items it defines, I agree with you, I just want the same consideration for ui generation.

nemesifier commented 7 years ago

Can we have a summary about the status of this idea? Is it stalled or is someone who has started working on it?

Anthropic commented 7 years ago

@nemesisdesign, the main focus is on the core and hyperschema draft updates. I am in email conversation with members, but it is slow while they must focus on their current task. If you have goals or interest in the outcome I'm happy to have a discussion through Gitter about it anytime :)

nemesifier commented 7 years ago

I'm the maintainer of a few OpenWISP modules that use JSON-Schema intensively (eg: netjsonconfig and django-netjsonconfig) and I also maintain the specification of the NetJSON data interchange format.

The interest in OpenWISP and NetJSON is growing rapidly, OpenWISP has also been accepted to the Google Summer of Code 2017. Yes I'm interested in contributing, I just need to organize myself a bit better. Probably when the GSOC ends I will have more time to dedicate to this idea. But in OpenWISP we are using a UI that is automatically generated from a JSON-schema and is working pretty well.

handrews commented 7 years ago

@Anthropic we're getting close to releasing the new drafts and there's really just the one big issue that we're finalizing, so now's probably a good time to get this going here, particularly if there is increasing interest :-)

nemesifier commented 7 years ago

Here's the main attributes that we are using heavily for UI generation, some of these are custom implementations of json-editor:

title
description
default: default value pre-filled in the UI
enum_titles - this one is custom, although I would prefer something like my old proposal choices, but this is better than nothing
{"format": "checkbox"}, this thing tells the UI generator to render as checkbox, it would be better to use a new attribute for this task, eg: widget
propertyOrder: this specifies the order of fields

handrews commented 7 years ago

It's occurred to me recently while working on doca (a documentation generation system) that there are quite a few features that would be shared by UI and API doc vocabularies.

For instance propertyOrder is more or less the solution @Relequestual is using for cloudflare/doca#28. And as much as I'd like doca to make a best effort to respect the file order and not have to re-specify all the property names, an interoperable solution would be better.

For enum_titles I still find the oneOf solution pretty compelling now that const is in the spec:

{
    "oneOf": [
        {"const": "foo", "title": "Pick Foo"},
        {"const": "bar", "title": "Pick Bar"},
        {"const": "whatever", "title": "Don't Care"}
    ]
}

This is an easy pattern to recognize, and it also allows associating other keywords like description and examples without having to come up with even more enum-enhancing keywords.

nemesifier commented 7 years ago

Honestly, if I had an alternative I wouldn't use that construct as an alternative to enum_titles. It's much more complicated and it has the negative side of being hard to read for someone who doesn't know JSON-Schema, and UI designers will much likely have to deal with UI schemas and I see this as a very likely adoption issue.

handrews commented 7 years ago

The advantage is that it actually works for both UI and validation. Being able to re-use existing keywords makes it both more flexible and more readable than the positional tuples for "choices", and avoids the duplication required by "enumNames".

Anyway, we can debate that specific issue when we get into the details. One solution might just be an alias for "oneOf" that is friendlier to UI usage so that we don't add much more implementation work for something that should already be present in implementations.

Anthropic commented 7 years ago

We use titleMaps which as you can see below replicate the exact structure of @handrews proposal with only a name change, the things we add though are allowing a group property and data related attributes so that a selection can be used to provide further information. So if the value needs to be the same as the name but a related select needs to be populated with a call using the selection's id then that is available.

{
    "titleMaps": [
        {"value": "foo", "name": "Pick Foo", "group": "Important Foo", "id": 4},
        {"value": "bar", "name": "Pick Bar", "group": "Important Foo", "id": 7},
        {"value": "whatever", "name": "Don't Care", "group": "Additional Options", "id": 42}
    ]
}

So ui:group and ui:data could be in a ui vocabulary to provide the ability to group related items in any context and also provide access to properties for related items as needed.

Anthropic commented 7 years ago

@nemesisdesign we currently use type in our ui-schema for widget, but I want to change it to ui:widget, so I agree that would be easier than using format which I think is preferred for specifying the data format than the ui widget.

brettz9 commented 7 years ago

Just adding mention of i18n (internationalization) / l10n (localization) here so this issue will turn up on searches for it (and in case participants haven't considered its relevance).

handrews commented 7 years ago

Thanks, @brettz9, it has come up but searchability is good.

brettz9 commented 7 years ago

With UI concerns now targeted to their own spec, I'd like to restate here for i18n that I think it is best for portability between consuming projects, for easier offline support, etc., to allow locales to be expressed in a standard way without need for server-side interpolations (allowing clients to substitute parameters that determine the choice of locale).

To i18nize the example from @handrews , something like the following could allow the documents to be distributed across project with internationalization information intact and consumable or switchable as desired:

{
    "definitions": {
        "locales": {
            "en-US": {
                "fooset": [
                    {"const": "foo", "title": "Pick Foo"},
                    {"const": "bar", "title": "Pick Bar"},
                    {"const": "whatever", "title": "Don't Care"}
                ]
            }
        }
    },
    "oneOf": {"$ref": "#/definitions/locales/~{locale}~/fooset"}
}

The locale variable could be substituted dynamically (including in JavaScript) after determining the user locale.

brettz9 commented 7 years ago

Regarding enum-enhancing keywords, for UI purposes, I'd like to see a means of indicating an equivalence between two sets of enums beyond l10n. While grouping and data attributes could be used to implement this, I don't think it would be ideal for making clear the nature of the relationship (equivalence).

For example, a JSON schema of a tabular version of the Bible (with columns for book, chapter, verse, text, annotations, etc.) might wish to express an equivalence for its "book" column between a long name ("Genesis", "Exodus", etc.) and well-known abbreviations ("Gen.", "Ex.", etc.). (Or, with the likes of the Qur'an, frequently-referenced Surah numbers could be mapped to Surah names, e.g., "1" or "The Opening".) More than one set of mappings could be desirable (e.g., those belonging to different sets, e.g., long, short, and medium-length: "Exodus", "Ex." or "Exod." OR those which happen to have idiosyncratic alternatives, e.g., "The Family of 'Imran" or "The House of 'Imran").

A generic book browsing tool could introspect on these equivalences and provide a choice of pull-downs and/or a text box which parses either set of abbreviations for a range of verses/paragraphs (e.g., "Gen 1:5-Exodous 2:7). Likewise, could the user opt for the output to display one set of aliases or another (e.g., to selectively list "Gen.", "Genesis", "Book 1", etc. alongside the chosen selection of verses). User agents could differ in how to present this, but they would have a standard way of detecting that alternative sets of equivalent labels were available.

romainpiel commented 7 years ago

(Sorry I might be deviating from the original topic)

I am wondering what would be the scope for a JSON UI Schema. Is the purpose to build a JSON specification for web UIs? Or should it be generic enough to also support web and native mobile UIs?
This thread (and https://github.com/json-schema-org/json-schema-spec/issues/252) only mention form UIs. Are there also motivations to consider more complex UIs with templates and constraints?

A bit of background aruond me and my motivations: I'm an Android developer and my day to day job involves converting data structures defined in a JSON payload to view models that could be easily bound to Android views. I'm currently doing some research for my personal interest to make my Android apps a bit more dumb and have our backend services expose data structure that are closer to view models, ready to be bound to the views. This sounds all great and I could just create my own JSON spec. But as I was starting to face issues like the one described in this thread, I thought maybe there could already be "UI-focused" specifications out there. If I'm not deviating too much and you folks need a pair of eyes from a mobile dev, I'd be happy to help.

Anthropic commented 7 years ago

@RomainPiel feel free to contribute to the JSON Schema UI vocabulary proposal based on this issue. I consider it to be more ui than form based myself, but I need more people to provide input like you have done for us to get some further traction on defining a solution and the limits of said solution.

handrews commented 7 years ago

@RomainPiel I agree with @Anthropic, who is the point person on this topic. We would really like to get critical mass for discussions so please do comment over on the vocabulary project!

romainpiel commented 7 years ago

@Anthropic @handrews great, will do!

handrews commented 7 years ago

This is now being tracked at json-schema-org/json-schema-vocabularies#2 as its own project.

Please join us there for further discussion!

json-schema-org / json-schema-spec

[Proposal] JSON UI Schema #67

252 2017-02-16