schemas: links & schemas, and the `&` hinting mechanism

[rvagg]

Some words about links and schemas to clear up some thinking on the topic.

Added the concept of a & hinting mechanism that has been floating around some of our discussions and I even used in the HashMap spec schema. I don't have a proposal for this would be represented in the reified schema form but I could imagine a "hints" block in addition to the "representation" block, but I could also imagine this filling up with other things useful for codegen.

Discuss ..

[warpfork]

This reads really well.

One big question... do we really want to have type Foo link be supported anymore? I think that was one of the earliest syntaxes, but it also predated any mechanisms of type hinting for the target, and now that we have the &TargetType syntax, do we want to have both?

I'd consider type Foo link to be the same as type Foo &Any. And I'd generally rather have One Way To Say Things, so if we make a format-normalizing tool for schemas (e.g., takes the json/ipld representation and prints out DSL again) it's clear what the tool should do.

On the other hand, I can see that together with the order you've presented it, type Foo link is much, much easier to gradually explain than the fish-slap-to-the-face of &Any. So I'm of mixed feels.

(We also still definitely have the characters link as a keyword in at least one place, regardless of the above -- kinded unions. For technicalities of parsers, that's clearly a different position in the grammar and easily handled, but for visual consistency it's also maybe something to consider at the same time as the above.)

[rvagg]

One big question... do we really want to have type Foo link be supported anymore?

Weakly held opinion: yes.

Further to your points, the bit about this being a "hinting" mechanism pushes schemas using & into new cross-block territory. In a sense, when you use &, even if you're doing &Any, you're making assertions about what you'll find on the other end of that link, which schemas can't do in their basic usage.

We're going to find other places where this is true as we want to make schemas more than just a node validation mechanism and it might be worth preserving a "pure" form of schemas that can do just that. You may want to even have a mode that enforces this: "sorry, you can't use an impure schema for protocol validation". ("nonlocal" might be a nicer name for that).

If anyone has stronger held opinions though .. ?

[warpfork]

Ohh. Okay. Yeah, that took me a second to figure out, but I see what you mean.

I guess I'm more on the syntactical level. I'm just worried about link vs &Any being very different syntaxes for essentially the mechanically same thing. I can see your point about &Any being sort of an assertion about the farside, but.... ehhh. Is a trivially-true assertion usefully distinct from no assertion in this situation? Will we actually do anything different between the two?

This might be be a situation where we should toggle back to thinking about what the matching schema-schema would look like as JSON, and see if that gives us any useful hints. I don't think the current drafts cover this: they only get as far as links, and don't have a defined place for the farside hint yet.

[rvagg]

So having my head in the Go parser made made me see some of this differently. I think at this stage I could live with &Any being the one way of saying Link but we could embed the softness of the "of type" assertion with what we call the member that records this type. I propose expectedType. So the JSON form of type Foo &Bar would be: "Foo": { "kind": "link", "expectedType": "Bar" }.

Outstanding questions:

• Does this one way apply to link as well as Link? So type Foo &Any is the new type Foo link? Just as type struct { Foo &Any } is the new type struct Foo Link }?
• Does &Any let you drop expectedType in JSON form because it's the default?

And ... we still need to define Any.

[rvagg]

pushed fcee7d8 which adds expectedType to TypeLink:

type TypeLink struct {
  expectedType Any
}

Also noticed there's a note in there suggesting as much, even calling it a "hint", so it's nice to see we're in sync on this.

I had a think about defining Any in there, it's already referenced in StructRepresentation_Map_FieldDetails#default, but does union of the base kinds work for that, like Type but without union, struct and enum? Or do we just need to describe it in the comments?

(Aside: the Type union is technically invalid, it's kinded but also includes union, struct and enum—and even the number kinds present difficulty if we want to be picky).

[warpfork]

Bikeshed idea: "expectedTarget"?

Does &Any let you drop expectedType in JSON form because it's the default?

Yeah, that's a good question. I'd say..... yes?

[warpfork]

Nah, forget that bikeshed.

This is awesome :+1:

[rvagg]

type Any union {
    | TypeBool bool
    | TypeString string
    | TypeBytes bytes
    | TypeInt int
    | TypeFloat float
    | TypeMap map
    | TypeList list
    | TypeLink link
} representation kinded

[warpfork]

I think we can still punt on 'Any' for this PR. We're mentioning it, but only in passing. Technically, in fact, only as a string (of type TypeTerm) here.

We should do it soon though, yeah. (StructRepresentation_Map_FieldDetails.implicit -- not touched in this diff -- already refers to Any and that's... tsk.)

[rvagg]

(reviewed and agreed while on a call with @warpfork)