Relating Type Params and Arg Usage

[harrysolovay]

The current shape and detail of FRAME metadata doesn't give us the full picture necessary to understand the relationships between type params and the fields that reference them.

The Option type is a good example.

Its params field looks as follows:

[
  {
    "name": "T",
    "type": 10
  }
]

And it's def looks as follows:

{
  // ...
  "members": [
    {
      "name": "None",
      "fields": [],
      "index": 0,
      "docs": []
    },
    {
      "name": "Some",
      "fields": [
        {
          "type": 10,
          "docs": []
        }
      ],
      "index": 1,
      "docs": []
    }
  ]
}

There is no way to determine that the first field of Some corresponds to the type param T. We can flatten the type according to whatever is at index 10... but we can't reconstruct an accurate generic representation. This poses great difficulty for generating idiomatic type definitions in other languages (as I'm currently trying to do in TypeScript). I'd greatly appreciate more information about the current approach.

[harrysolovay]

Here is a an example of the "overloaded" option type––with the type param constraint set to a union of those output during monomorpization.

... there are more members of the constraint union, but they would not fit on the screen.

To take this approach to modeling Option in TypeScript, we need to parameterize each variant according to which args they make use of. In other words, we must figure out if/how to parameterize the Some and None types. What type params do we add to the following?

A few changes need to be made to this code sample. Some needs the same type param as Option. To figure this out, we'd need to visit the variant's fields and determine which param names are referenced (we'd also swap out field 0 with the T that reference). Unfortunately, we lack the information necessary to do so.

---

Regarding alternative routes...

We could forgo undoing the monomorphization and generate constraint-specific versions of each type.

export namespace Option {
  export type _numberLs<T extends number[]> = Some._numberLs<T> | None;

  export type _ElectionCompute_1<T extends ElectionCompute_1> = Some._ElectionCompute_1<T> | None

  export type _ScheduleV3_1<T extends ScheduleV3_1> = Some._ScheduleV3_1<T> | None

  // misc ...
}

... but this is quite bloated.

Alternatively... we could forgo the type param constraint entirely. Given that this will be generated, we don't have to worry about users supplying invalid types (the user never directly instantiates these types). Still, we'd run into the issue of not knowing which fields/types within those fields reference which type param names. And even if we could get this to work, it's not ideal for discoverability––developers should be able to understand what types are valid in a given slot.

---

Any thoughts would be greatly appreciated!

[ascjones]

Capturing generics is a hard problem, I did some work on it which I abandoned https://github.com/paritytech/scale-info/pull/12, in order to prioritize integrating the new metadata into Substrate.

The workaround for arbitrary generic types is to use the optional typeName on the field to match up with name field from the params. That way you can substitute the concrete type for the generic type where it belongs.

[
  {
    "name": "T",
    "type": 10
  }
]

{
  // ...
  "members": [
    {
      "name": "Foo",
      "fields": [
        {
          "type": 10,
          "typeName": "T",
          "docs": []
        }
      ],
      "index": 1,
      "docs": []
    }
  ]
}

For all types where TypeInfo is derived the typeName field should have a value. However where it is not the case you can use the concrete type to match the field type with the type parameter to substitute it as T.

Option is a built-in type so has a manual TypeInfo implementation which doesn't have the typeName set: https://github.com/paritytech/scale-info/blob/master/src/impls.rs#L199. Arguably it should do, however because it is a "built-in" type that will never change we can make the assumption that the first field of the Some variant always corresponds to the only type parameter. In the subxt Rust codegen we just use the library Option type instead of generating our own version: https://github.com/paritytech/subxt/blob/08369f3e4399fee54bb5e89bececf8285968d554/codegen/src/types/type_path.rs#L119.

So for the specific case of Option and other built-in types e.g. Result it might be good to define your own local hardcoded copies. Might be worth having a look at polkadot.js codegen to see what they do there.

It may be possible in the future to come up with a nice way to represent generic types in the metadata itself, but that is not likely in the near future, and at the moment the above described heuristic should be good enough.

[harrysolovay]

Thank you for clarifying that typeName isn't present on those built-in types (Option, Result). In other cases, typeName should give us what we need!