Nil and required differences could be ambiguous and hard to understand

[boillodmanuel]

Hello,

Today, RAML (And json schema too) distinguish two concepts:

• required: should the property be present in the payload, whatever the value
• nil: could the property value be null

These two concepts are a bit technical and for end-users, the question is often limited to "should I set a value or not".

The difference between property missing or property present with null value is often a serialization / technical problem.

I've been working with Swagger and RAML for a long time now and always had to read the doc several times to be sure of my understanding, and actually, I'm never sure :) When I asked my team's members, it turned out this was confusing for them too.

So, I will try to summarize what I understand here.

Required value

In my examples, I will use the following Person type and focus on surname attribute:

types:
  Person:
    properties:
      name:
        type: string
      surname:
        type: string

With this definition, the description property should be present and its value should not be null, ie:

• { "name": "Mickael", "surname": "Mike" } is VALID
• { "name": "Mickael", "surname": null } is NOT valid
• { "name": "Mickael"} is NOT valid

  Optional value

If a user cannot fill the surname field (in a form for example), we should use this RAML definition:

types:
  Person:
    properties:
      name:
        type: string
      surname?:
        type: string?

or the more verbose one:

types:
  Person:
    properties:
      name:
        type: string
      surname:
        type: string | nil
        required: false

In this case, we have:

• { "name": "Mickael", "surname": "Mike" } is VALID
• { "name": "Mickael", "surname": null } is VALID
• { "name": "Mickael"} is VALID

IMHO when we are talking about an optional property (at business level), this is what we have in mind.

So there are two interesting/common cases:

1. required and type not containing nil
2. not required or type containing nil.

Most of the people I talk about this issue with think required attribute means option 1 and not option 2 and don't ever use type: nil anywhere. They don't consider the remaining options (required but can be nil, not required but cannot be nil) because not specifying a property is most of the time treated as if the result was there but nil. The difference only is in the serializer's configuration.

I wonder if there is a real demand/need to distinguish other options?

By the way, I think that developers should use the required: false and type: string | nil to define optional property (always at business level). I don't thing the developers are not doing this, may be due to bad understanding of the spec. Am I right?

If true, I think it might be a good idea to remove this ambiguity in the documentation. What do you think?

Optional request body

In the same way, we wonder how describe an request that have an optional body (not very common). Is it supported by the RAML specification? Is the following declaration correct?

/persons/{id}:
  get:
    responses:
      200:
        body:
          application/json:
            type: Contact | nil

or with the syntactic sugar: type: Contact?.

Hope I was clear.

Manuel

[freddrake]

Not sure how best to deal with an optional request entity, but an omitted entity would no longer be a valid JSON document (or XML), so would not be found under body: application/json:.

The example given would lead me to consider an application/json body containing the text null to be valid, but that's not an omitted request entity.

The difficulty in explaining the difference between a specified null and an omitted property always surprises me, but it not specific to RAML. An example or two should be sufficient to explain what the difference is. Since RAML is intended as a way to describe what's valid for a service, being able to spell all these flavors is important, because it should be possible to describe an existing service.

[boillodmanuel]

I agree that required is better than Contact | null for a request body and also that an optional request entity "would not be found under body: application/json" But currently, this doesn't exist in RAML.

The difficulty in explaining the difference between a specified null and an omitted property always surprises me,

I guess this is too technical. People didn't ask themselves if a missing property is equivalent or not to as null property. They just say that "phone number" is an optional attribute of a "User".

So, do you confirm that an optional value should be described like this (if we don't take care of property existence):

User:
  properties:
    phoneNumer: 
      required: false
      type: string | nil

or in the simpler form:

User:
  properties:
    phoneNumer?: string?

[freddrake]

I guess this is too technical. People didn't ask themselves if a missing property is equivalent or not to as null property. They just say that "phone number" is an optional attribute of a "User".

Were we referring to arbitrary people selected off the street, "too technical" would be a reasonable understanding of the confusion. For someone who's a programmer, whether front-end or back-end, I'd say they need to be educated.

Some people may consider me harsh.

So, do you confirm that an optional value should be described like this (if we don't take care of property existence):

If I expect a User may have a phone number, I'd expect to receive in a POST or PUT request:

User:
  properties:
    phoneNumer?: string

In an implementation, either I get one, or I don't.

For a PATCH request, I'd use something like this:

UserUpdate:
  properties:
    phoneNumer?: string?

If phoneNumber is omitted from the request, I'd not change the stored data, but if null, I'd clear any stored phoneNumber value.

[boillodmanuel]

You're not too harsh. It's fine :) But I always keep in mind that RAML is to be written by technical people and intend to provide tools like documentation for all kind of people, including non technical (support, QA, product owner, ...)

I understand the distinction you did between POST and PATCH cases, and it make sense. But technically it could be difficult. For example, in Java (as well as in Obj C, and others), distinguishing null property or missing property is very hard. No library provide that off the shelve. So if you really want that feature, you should write it yourself. And for the PATCH method, missing property could fit for very simple case, but when you want a more robust and general solution, you should move to JSON PATCH or something like that.

This is the main reasons which make me think that phoneNumer?: string? is the most often wanted and the doc should warn the user of this subtlety.

I let you decide what to do next (may be just updating the doc).

[freddrake]

But I always keep in mind that RAML is to be written by technical people and intend to provide tools like documentation for all kind of people, including non technical (support, QA, product owner, ...)

The only people who need to understand everything in a RAML specification are technical people. Non-technical people will stop at the natural-language documentation, or maybe the list of properties (just the names, nothing more).

Perhaps other organizations are different, but I've never found a product owner who wanted to be that deep in the technology behind their products. Maybe I'm just missing out on the fun?

I understand the distinction you did between POST and PATCH cases, and it make sense. But technically it could be difficult. For example, in Java (as well as in Obj C, and others), distinguishing null property or missing property is very hard. No library provide that off the shelve. So if you really want that feature, you should write it yourself.

I absolutely believe that this isn't something commonly expressed in libraries provided for many languages, especially where the notion of a property only sometimes being present on an instance of a class is common. The notion of a optional property is not universal, and support generally requires selection of completely different data structures.

And for the PATCH method, missing property could fit for very simple case, but when you want a more robust and general solution, you should move to JSON PATCH or something like that.

By JSON PATCH, I presume you're referring to https://tools.ietf.org/html/rfc6902 (http://jsonpatch.com/). That's interesting; I don't recall seeing that before. Of course, that defines a JSON-to-JSON transform, so an implementation probably needs to be available for each data-storage back-end, with augmentation for "interesting" data structures.

This is the main reasons which make me think that phoneNumer?: string? is the most often wanted and the doc should warn the user of this subtlety.

It may make sense to add language to point out that this is a commonly desired pattern, but I think it should be very clear about what it's used to allow. I definitely think it's important to be able to spell the separate aspects of this, as those are fundamental, and this is simply a (possibly common) usage pattern.

I let you decide what to do next (may be just updating the doc).

I don't think I'm a decision maker on this myself.

[sichvoge]

@freddrake @boillodmanuel sorry for being on that a bit late. what i get out of your conversation is that the current documentation around the nil type is not sufficient enough to completely understand what and when it should be used, perhaps through examples like in @boillodmanuel OP. Is my understanding correct?

I am really happy to review and merge any PR that extends the doc with more clarification. Lets work together to improve the content and bring it to the next level before we start defining 1.x. ;)

[freddrake]

@sichvoge My point is that additional clarification isn't required, since the idiomatic usage @boillodmanuel describes is just that, and someone writing an API specification in RAML will get it right if they understand RAML.

I do concede that it may be easier for new software engineers to pick up on the idiomatic usage if there's additional non-normative language describing the idiom explicitly. Specific language should probably be tested on real newbies to ensure it actually helps. (And I'm not at all sure how to go about that.)

[boillodmanuel]

Actually, there are 2 things 1) It seems there is no way to define an optional request body (a bit out of the scope of the title of this issue) 2) This is very difficult to well understand the subtlety of required and nil. And I disagree with @freddrake on this topic. I think noob and advanced RAML user (as for JSON schema) don't understand well this subtlety, Expert users do. And in my opinion, most of users don't want to distinguish required and nil value, thus, they mostly want name?: { type: string?} and they often use only required. But I have no much time to do a PR to improve the doc, I'm sorry.

[p-bakker]

In case of GETs, you might want document that the property will always be returned, but might be null, as in: name: string?, so suggesting that people mostly want to use name?: string? is not correct., it depends on the context

On PATCH/POST/PUT, you probably want to use either name?: string? or name: string depending on whether the name is required or not from a business logic perspective. Using name?: string? is the name is not required gives the user of your API the ability to either specify the property with a null value OR not specify the property at all, which imho is good DevUX

[tonedef71]

Is it possible to define a property of type string that may be null, but when not null may still be constrained by the other string-type properties (i.e. minLength, maxLength, pattern)?

e.g.

email:
  type: string | nil
  minLength: 6
  maxLength: 250
  pattern: ^\w+([-+.']|\w+)*@\w+([-.]\w+)*\.(\w+([-.]\w+)*){2,}$
  example: a+d@b.cc

[jstoiko]

@tonedef71: this is invalid as it stands because it would effectively apply minLength to the nil type. However, you can achieve this by “externalizing” the type you want to be nillable. E.g.

types:
  Email:
    minLength: 6
    maxLength: 250
    pattern: ^\w+([-+.']|\w+)*@\w+([-.]\w+)*\.(\w+([-.]\w+)*){2,}$
    example: a+d@b.cc

then:

email:
  type: Email | nil

[caiblack]

I’m not sure that I agree with your assessment.

This says “type is either Email or nil”.

email: type: Email | nil

I don’t see any ambiguity there. The minLength and maxLength attributes in the Email type definition do not influence the handling of nil.

Caveat: unless I’m mistaken....

[sent from my iPhone]

Christopher (Cai) Black caiblack@hotmail.commailto:caiblack@hotmail.com 832-439-8134 Web Application Architect Visual Effects (VFX) Artist

On Dec 14, 2018, at 11:23 AM, Jonathan Stoikovitch notifications@github.com<mailto:notifications@github.com> wrote:

@tonedef71https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftonedef71&data=02%7C01%7C%7Ce0f447171483410a02b408d661e8d224%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636804049909008005&sdata=mGlfjtZgoAgd3XmEWMourtZPNKM%2BSLSRh8uiNoXg0gY%3D&reserved=0: this is invalid as it stands because it would effectively apply minLength to the nil type. However, you can achieve this by “externalizing” the type you want to be nillable. E.g.

types: Email: minLength: 6 maxLength: 250 pattern: ^\w+([-+.']|\w+)@\w+([-.]\w+).(\w+([-.]\w+)*){2,}$ example: a+d@b.ccmailto:a+d@b.cc

then:

email: type: Email | nil

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Framl-org%2Framl-spec%2Fissues%2F569%23issuecomment-447393164&data=02%7C01%7C%7Ce0f447171483410a02b408d661e8d224%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636804049909008005&sdata=rjHxIFzC1doTexoELrq%2FIuSJUZ%2FDsWscofp%2BIuQxNdY%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAM9c9BVMt_vIigYaNUgp0amwDyQAtjXks5u4959gaJpZM4J36VW&data=02%7C01%7C%7Ce0f447171483410a02b408d661e8d224%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636804049909008005&sdata=yNErp4JAt%2FVpxqn7oIKAQiyGM6ATIboGBCSeYxDqBTk%3D&reserved=0.

[jstoiko]

minLength and maxLength are not allowed facets for the Nil type. The Spec explicitly defines all allowed facets for each type.

This could however be further clarified in the Union type section of the Spec. I'll suggest some clarification in the next few weeks in the form of a PR and will circle back here.