Null primitive type

[netmilk]

In this Dredd issue is @neonichu having a body JSON key with null value. There is no way how to express it in MSON at this moment. Can you please add support for null primitive type in MSON?

[itsjamie]

So, wouldn't omitempty be a superset of optional?

Optional allows a key to be omitted from the resulting JSON, similarly omitempty allows a key to be omitted, but would fail if the key was present with its "empty" value.

In Go, which this suggestions seems to have been influenced by you are looking at these rules in the JSON package:

The empty values are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero.

Which essentially map this way..

JSON Types	Empty Value For Omission	Go Type That Encodes To Value
Array	[]	[]slice
Object	{}	struct{}
Number	0	int/float..
String	""	string
Bool	false	bool
Null	null	nil pointer

[obihann]

@zdne null as a primitive type is not hiding the true type, because null is a valid JSON type along with number, bool, and string. Allowing null in place of another object is only part of the issue with MSON, the bigger issue in my opinion is the lack of null as a type.

[zdne]

@obihann null may be a type in JSON but no so in many other programming languages or serialization formats. As such it does not convey any information about the type of the value besides it is "not set".

[zdne]

As discussed in https://github.com/apiaryio/mson/pull/35#issuecomment-134181674

I would like to come up with solution for following:

1. allow a value of any type to be null (not set) while retaining information about the actual type of the value.
2. allow a value of any type to be a default empty value ("", [], {}, 0, etc.)

Does everyone involved here agree that solving this will be solution for his/her respective problems?

[obihann]

@zdne Good point, I keep look at this from the issue if null as a value, not null as a type and what that may or may not represent.

And yes I believe we need:

1. Not set: null
2. Empty: "", 0, non-null

Something that may need to have some thought though, is 0 truly empty? If one wants an optional number field would 0 be representative of a empty value or an actual 0 value? Perhaps that is not really relevant here.

[zdne]

Going back to null as a allowed value...

1. Expressing "no value" and "empty value"

MSON

- key1: null (string)
- key2 (string)

JSON

{
    "key1": null,
    "key2": ""
}

✅

{ }

✅

Swift

let key1: String?
let key2 = String()

✅

2. Omit Empty

By default, optional values with null or default values MAY but DOESN'T HAVE TO be ommitted. To force "omit empty" use omitempty type modifier.

MSON

- key1: null (string, omitempty)
- key2 (string, omitempty)

JSON

{
    "key1": null,
    "key2": ""
}

❌

{ }

✅

3. Expressing "no type" or "any type"

A type placeholder – generics support.

MSON

- key1 (any)

JSON

{
    "key1": null
}

✅

Swift

var key1: AnyObject

✅

NOTE: The generics sections of specification needs tighten up.

---

Thoughts?

[obihann]

Looking at the first MSON example:

- key1: null (string)
- key2 (string)

I like it, however it seems to me this would mean by default we allow null, is this your thought? If so can specify a rule to inforce no null, or would required handle that?

--edt--

On second thought I don't believe required being used for non-null is a good idea, required is for when a field has to exist, which is very different than non null in therms of schema.

[zdne]

@obihann

by default we allow null, is this your thought?

No, default is the default (erm) empty value – that is "", {}, [] etc.

If you would like to override it I would suggest

- key1: Hello! (string)
    - default: null

Which is equivalent to

- key1 (string)
    - default: null
    - sample: Hello!

[zdne]

Which brings me to...

- key (any)

For the hypothetical any type, in JSON representation the default value would be null. Hmm.

So maybe we do have it?

That is

MSON

- key (any)

JSON

{ "key": null }

I guess it is all just one big mental exercise for me (and hopefully others) in that null type is just "any" type.

[obihann]

Sorry, I didn't mean default in the sence of a default value. What I was trying to say was are we accepting null as an acceptable value without any additional flags? Here is an example of what I'm trying to say:

Option A

null is ALWAYS accepted as a value for any type

MSON

- key (string)

JSON

Accepted

{"key": "cat"}

Also Accepted

{"key": null}

Option B

null is accepted as a value for any type WHEN notset is applied

MSON

- keyA (string, notset)
- keyB (string)

JSON

Accepted

{"keyA": "cat"}
{"keyB": "cat"}

Also Accepted

{"keyA": null}
{"keyB": "cat"}

Not Accepted

{"keyA": null}
{"keyB": null}

[zdne]

I think your "option B" is what was discussed above with nullable type modifier.

Please disregard my previous comments on "any type", albeit we may need it for generics in this scenario it is not applicable. Too late I've realized that its equivalent in JSON world would be all 7 primitives types (from the JSON Schema perspective).

So I feel like we are back in square one :)

[obihann]

Based on all the suggestions, heres what I suggest:

2.1.1 Primitive Types

Applies to basic data structure and type definitions. [Types][] that are sub-typed from Primitive Types MUST NOT contain a [Member Type Group][] or [Nested Member Types][].

• null

  Specifies a null type.

• boolean

  Specifies a type with allowed values of true or false.

• string

  Specifies any string.

• number

  Specifies any number.

3.5.3 Type Attribute

Defines extra attributes associated with the implementation of a type.

• nullable - The value may include {}, null, or []
• required - instance of this type is required
• optional - instance of this type is optional (default)
• fixed - instance of this type structure and values are fixed
• sample - Alternate way to indicate a [Value][] is a sample. See [Sample][].
• default - Alternate way to indicate a [Value][] is a default. See [Default][].

A sample Type Attribute is mutually exclusive with default.

Update: Removing null as a type, as @pksunkara pointed out with the addition of a nullable rule it is not needed

[pksunkara]

If we are using nullable type attribute, we don't need a null Type Attribute.

[obihann]

@pksunkara great point, I'll remove that!

[hobofan]

nullable - The value may include {}, null, or []

I think that this would be misleading and wouldn't allow to differentiate between null and {} unless you limit it to their respective "notset" value:

• string, boolean, number -> null
• Array[xxx] -> []
• Object -> {}

Option A

nullable would mean "not-set" or "empty" value of their respective type (I wouldn't go with nullable as name.)

MSON

- keyA (string, nullable)
- keyB (Array[string], nullable)

JSON

Accepted

{"keyA": "cat"}
{"keyB":  ["cat", "dog"]}

Also accepted

{"keyA": null}
{"keyB": []}

Not Accepted

{"keyA": null}
{"keyB": null}

Option B

null is an existing type nullable would mean "empty" value of their respective type only for Array and Object (I wouldn't go with nullable as name.)

MSON

- keyA (Enum[string, null])
- keyB (Array[string], nullable)

JSON

Accepted

{"keyA": "cat"}
{"keyB":  ["cat", "dog"]}

Also accepted

{"keyA": null}
{"keyB": []}

Not Accepted

{"keyA": null}
{"keyB": null}

I think option B would better capture the intention to both provide a way to specifiy possible null values and to allow "empty" values for Arrays and Objects.

Update:

Option B would also allow:

MSON

- keyA (Array[Enum[string, null]], nullable)

JSON

Accepted

{"keyA":  ["cat", "dog"]}

Also accepted

{"keyA": []}

Accepted; This one would be hard without a actual null type

{"keyA": [null]}

[obihann]

Good point, perhaps we need to define nullable and empty, however would empty to better differentiate between things like [] or {} and null. What about this:

nullable - The value may be null allowempty - The value may be {}, [], 0, or ""

I'm starting to think we don't need a primitive type null because it would likely never be used outside of an enum to allow the additional value, I can't see this ever being required:

MSON

- key null (null, required)

[hobofan]

@obihann : Did you take my update into account? I agree that there probably won't be much use outside enums, but I think that the combination with enums alone is very powerful and desirable on its own.

[obihann]

@hobofan I may have missed it, so your feeling is that we use null as a primitive, and nullable as an option to define something like [null]?

So if I wanted a string with that allows null I would do enum(string, null) however if I want an array that allows null I would just do array, nullable, and it would all look like this:

MSON

- keyA null (enum[string, null], required) - accepts strings or nulls
- keyB [null] (array, nullable, required) - accepts arrays or arrays containing null
- keyC null (enum[array, null], required) - accepts arrays or null

JSON

{
"keyA": null,
"keyB": [null],
"keyC": null
}

[obihann]

I agree the combination of a null type and a nullable option is a very powerful combination, however as you mentioned using nullable as a way to define [null] as an acceptable value is not straight forward to me. Perhaps allowempty would be best and then it could be like this:

allowempty - The value may be {}, [], [null],0,or""`

[hobofan]

Sorry for the misunderstanding. I meant to say that something like allowempty should be used to express {}, [], 0, "" and that Arrays like [null] and [null, "cat", null] should be able to be expressed by something like: - keyA: (Array[enum[string, null]], required) - accepts Arrays whose items are either strings or nulls

[obihann]

Ahhh, that makes sense. So this is where we are at then as I understand:

• the addition of a null primitive the addition of anallowempty` option

2.1.1 Primitive Types

Applies to basic data structure and type definitions. [Types][] that are sub-typed from Primitive Types MUST NOT contain a [Member Type Group][] or [Nested Member Types][].

• null

  Specifies a null type.

• boolean

  Specifies a type with allowed values of true or false.

• string

  Specifies any string.

• number

  Specifies any number.

3.5.3 Type Attribute

Defines extra attributes associated with the implementation of a type.

• allowempty - The value may include {}, 0, '""', or []
• required - instance of this type is required
• optional - instance of this type is optional (default)
• fixed - instance of this type structure and values are fixed
• sample - Alternate way to indicate a [Value][] is a sample. See [Sample][].
• default - Alternate way to indicate a [Value][] is a default. See [Default][].

Examples

MSON

- keyA (enum[string, null], required) - Accepts `null` or `string`
- keyB (enum[array[string], null], required) Accepts `null` or array of **only** `string`
- keyC (array[enum[string, null], required) Accepts an array of `null` or `string`
- KeyD (object, allowempty, required) - Accepts a populated or empty (`{}`) object
- KeyE (string, allowempty, required) - Accepts a populated or empty (`{}`) object

JSON

Accepted:

{
"keyA": null,
"keyB": null,
"keyC": [null],
"keyD":{},
"KeyE": ""
}

or

{
"keyA": "jeff",
"keyB": ["jeff"],
"keyC": ["jeff"],
"keyD":{"a":123},
"KeyE": "jeff"
}

Incorrect:

{
"keyB": [null],
"keyC": null,
"keyD": null,
"KeyE": null
}

[hobofan]

@obihann +1

[jrep]

On Aug 24, 2015, at 5:48 AM, Z notifications@github.com wrote:

I would like to come up with solution for following: allow a value of any type to be null (not set) while retaining information about the actual type of the value. allow a value of any type to be a default empty value ("", [], {}, 0) I think there are three states -- four, really:

1. not even present in a given instance of the object type (an "optional", typed field)
2. present, but set to null, with full information on what the type would be if there were a value (a "nullable", typed field)
3. present, but set to a type-appropriate empty value (a "defaultable", typed field)
4. and of course the boring, ordinary "set to some actual value of appropriate type"

Perhaps you were assuming #1 (optional) and #4 (actual), since they're already well-specified?

In the null case, I'm not certain where this type information would be retained. The MSON document itself (a document written in MSON; not the spec, written about MSON) should specify the type (like "(string, optional, nullable, defaultable)"). Were you also implying that the serialized form (JSON, XML, etc.) would carry some type info?

And I think maybe the "retained type info" problem exists even in some default-empty-value cases. Certainly it does in more pedantic languages, like Java, that can force you to type-cast a value before you're even allowed to check nullity. But if "actual type retention" applies only to the MSON itself, then the example immediately above covers it.

Jack Repenning Jack@netgate.net

[obihann]

@hobofan I've created a pull request with the latest changes, let me know if you still think its a good idea lol.

[obihann]

@jrep I believe null should be a primitive type, not a option such as nullable. This is because a) (as you stated) in JSON null does not reflect the alternative (string, object, etc) type that could be used as a value, and b) null is itself a valid type both from virtually every programming language, and JSON itself.

Also regarding the omitance of optional and actual, you are correct this is because they are already defined.

[pksunkara]

I think there are three states -- four, really:

1. not even present in a given instance of the object type (an "optional", typed field)
2. present, but set to null, with full information on what the type would be if there were a value (a "nullable", typed field)
3. present, but set to a type-appropriate empty value (a "defaultable", typed field)
4. and of course the boring, ordinary "set to some actual value of appropriate type"

We need to make sure MSON can be used to render the above four states.

But I agree with @zdne's comment that we shouldn't lose the information about the actual type. But that leads to the question whether @zdne meant it in the MSON or the rendered JSON/XML?

@obihann The problem with having a null type is that since MSON is just a modelling language, it is not tied to JSON directly. Yes, XML contains a method to represent null too, but it is not definitely an actual type. This is why I am completely against null primitive type.

If we can use the tags like nullable to represent the above 4 states, we can find a solution to this problem.

[obihann]

@pksunkara The very description of MSON is that is a markdown language for describing JSON and JSON Schema, if you read the specs for JSON schema section 3.5 you will see null is a primitive type. If you are choosing to ignore this fact and steer MSON into a different direction then fine, however the very description of MSON should be changed to reflect this.

[honzajavorek]

@obihann In MSON's README I see

Markdown Syntax for Object Notation (MSON), a Markdown syntax compatible with describing JSON and JSON Schema.

and the very second sentence is

MSON is a plain-text, human and machine readable, description format for describing data structures in common markup formats such as JSON, XML or YAML.

From my POV, MSON is definitely trying to be suitable for describing JSON and JSON Schema, but it has much higher ambitions than to be just a syntax sugar for those two.

[obihann]

If the only way we can resolve this issue is to use a nullable option and not a null primitive then I will gladly outline the new options, and submit a pull request for it, I don't want to delay this any longer as the lack of null support be it either a primitive or a option is hindering a project I am currently working on.

That being said, I would much prefer a null primitive.

[obihann]

Just to see if were on the same page, this is what I'm thinking:

3.5.3 Type Attribute

Defines extra attributes associated with the implementation of a type.

• nullable - The value may be null
• allowempty - The value may include {}, 0, ", or []
• required - instance of this type is required
• optional - instance of this type is optional (default)
• fixed - instance of this type structure and values are fixed
• sample - Alternate way to indicate a [Value][] is a sample. See [Sample][].
• default - Alternate way to indicate a [Value][] is a default. See [Default][].

Examples

MSON

- keyA (string, nullable, required) - Accepts `null` or `string`
- keyB (array, nullable, required) Accepts `null` or array of **only** `string`
- keyC (array[enum], required) Accepts an array of `null` or `string`
   - (string, nullable, required)
- KeyD (object, allowempty, required) - Accepts a populated or empty (`{}`) object
- KeyE (string, allowempty, required) - Accepts a populated or empty (`{}`) object

JSON

Accepted:

{
"keyA": null,
"keyB": null,
"keyC": [null],
"keyD":{},
"KeyE": ""
}

or

{
"keyA": "jeff",
"keyB": ["jeff"],
"keyC": ["jeff"],
"keyD":{"a":123},
"KeyE": "jeff"
}

Incorrect:

{
"keyB": [null],
"keyC": null,
"keyD": null,
"KeyE": null
}

[obihann]

I am holding off on creating another pull request however I do have a branch with the above attributes with examples available and ready

[pksunkara]

@obihann Thanks for all the work you are doing. But please do hold off on creating the PR since this discussion is not yet finalised. :smile:

[obihann]

@pksunkara no problem, however at some point we are going to have to make a discussion since this conversation has circled several times, and the author @netmilk has not been a part of the conversation in quite a while.

Since you are actually a contributor and a member of the apiaryio organization, how do you feel about the current suggestion?

[pksunkara]

@obihann The current solution does look good but we need to think of a few things before committing to it. MSON can be quite complex and we need to think of all the scenarios. I feel like we need to consider some related things which haven't entered into the discussion like Samples and Default.

As I said earlier, this problem is not as simple as it looks.

[obihann]

@pksunkara I see Samples being larger of an issue than Default, specifically because Default really only gives you one option. Based on that the following could be written:

MSON

- name: null (string, nullable, required) - Name can be a string or null, default is null

JSON Schema

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "",
  "type": "object",
  "properties": {
    "name": {
      "id": "/name",
       "anyOf": [
             {"type": "string"},
             {"type": "null"}
       ]
    }
  },
  "required": [
    "name"
  ]
}

JSON

Default:

{
   "name": null
}

Also accepted:

{
   "name": "jeff"
}

[obihann]

Samples shouldn't be much more difficult, the fact that you cannot combine a Sample with a Default might even simplify it further.

MSON

- name (string, nullable, required) - Name can be a string or null, default is null
    - Sample
        - null
        - "jeff"
        - "bob smith"

I believe the the Optional attribute would be unaffected, and Fixed would be very similar to the above example for Sample.

[zdne]

Lots of interesting discussion going on here!

@obihann thanks for leading the push. Do not worry about @netmilk – I am sure he is happy :smile: (as he is on vacation ATM)

[zdne]

So. I think one of the problems here is we do not have clearly described the problem and trying to solved to many things at once. Let me try to summarize what is going on here, and lets first agree on that.

[zdne]

MSON and null

Original Problem

@neonichu having a body JSON key with null value. There is no way how to express it in MSON at this moment. Can you please add support for null primitive type in MSON?

Rephrased Problem

How to describe a member type with null value?

Straightforward Solution

Introduce null as a possible value for any sample.

- key: null (string)

Drawbacks

Introduction of a new keyword may affect existing MSON documents.

Renderer Implementations

JavaScript, JSON, YAML

A null value would translate into the null TYPE.

XML

A null value would translate into an element with xsi:nil="true" ATTRIBUTE.

Swift, GO

A null value translates to the nil VALUE.

C++, C, PHP, SQL, JAVA

A null value translates to NULL (or 0) VALUE.

Python

A null value translates to None VALUE of NoneType TYPE.

Ruby

A null value translates to nil VALUE of NilClass TYPE.

PERL

A null value translates to undef VALUE.

Related Issues

This – seemingly straightforward issue – brings many related issues on the table.

1. How to distinguish between "empty" and "null" values

{ key: null }

vs.

{ key: "" }

2. How to write MSON for languages with and without a "null" type

3. How to control rendering

Of:

1. Empty values
2. Null values
3. Omitted key/values

NOTE: By rendering is meant representing what was described in MSON in the respective language / serialization format.

4. How to represent "any" type

[zdne]

Before we look at possible solutions – are there any objections against what is written in https://github.com/apiaryio/mson/issues/26#issuecomment-134632896 ?

[obihann]

None whatsoever, you hit the nail on the head when you said:

Rephrased Problem

How to describe a member type with null value?

[zdne]

Good! We do not have to, necessarily, solve all the related problems here. If we focus on the main thing. However, it will be good to keep them on mind as we are trying to solve the main issue so we wont block or complicate the solution of the related issues later.

[obihann]

We could focus specifically on null in this issue, resolve that, then create additional issues for the other issues / ideas raised. I would be glad to help work through all of them. I fell like family here now :P

[zdne]

He he. Agree. Let's focus on the null. First without closing paths to the other issues.

Focus on the Original Problem

So hypothetically, if we do introduce null as a keyword and a possible value:

- key: null (string)

What needs to be solved?

1. Rendering in JSON Schema

Does the example render as

{ "type": [ "string", "null" ] }

or

{ "type": "string"}

or

{ "type": "null"}

2. Rendering in JSON

Does the example render as (remember it is optional in the example)

{ "key": null } 

or

{ } 

NOTE: If you would like null as a text you would have to backtick escape it - key:null(string).

[jrep]

Sorry, I must have blinked: what is "fixed"? (Update: for the record, I guess I meant "what is fixed?". But both forms of the question are now answered;-)

[obihann]

@zdne I believe the correct JSON schema would be something like this:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "",
  "type": "object",
  "properties": {
    "name": {
      "id": "/name",
       "anyOf": [
             {"type": "string"},
             {"type": "null"}
       ]
    }
  },
  "required": [
    "name"
  ]
}

Regarding 2, I'm not a huge fan of {} === null I think that is a good place for an allowempty attribute however as we discussed we should focus on null only, so I think we just ignore empty values for now as the introduction of null shouldn't directly effect them.

All that being said @pksunkara is not a fan of null as a primitive type and though it is my preference, I am comfortable with a nullable attribute.

[obihann]

@jrep Nothing really was fixed, @zdne and I just felt that the conversations of things like empty values should be left out of this issue because they are not directly related to the original issue, the lack of null support.

Mostly we are trying to circle back to the beginning and resolve what was originally reported as an issue.

[zdne]

@jrep all the values are considered samples. If you have fixed a thing it means it has to appear exactly as it was written.

Consider array of strings with two sample values

- key (array[string])
    - one
    - two

vs. a tuple (pair) that always has "one" and "two" literals in it

- key (array[string], fixed)
    - one
    - two

[zdne]

fixed also fixes the order and count so

- key (array, fixed)
   - (string)
   - (number)

says the array is a pair of a string and a number in this order.

[zdne]

ah, fixed or fixed, bounded context ;)