feat: `@type` intrinsic

Chriscbr commented 2 months ago

Closes #7150

Adds for the new @type intrinsic function. This function can be passed a type, and it will give you back an object (std.reflect.Type) with information about it -- for example, whether it's a struct or class, what properties or fields it has, etc. You can use this to validate request data structures at runtime, generate schemas for APIs or databases, or for simply asserting on the structure of your code.

Future work:

Support doc reflection
Fix support for variadic functions
Open discussion about accessing private fields or methods via reflection
Open issue about explicit "preflight" annotation
Open issue about explicit TypeId enum

Checklist

[x] Title matches Winglang's style guide
[x] Description explains motivation and solution
[x] Tests added (always)
[x] Docs updated (only required for features)
[x] Added pr/e2e-full label if this feature requires end-to-end testing

By submitting this pull request, I confirm that my contribution is made under the terms of the Wing Cloud Contribution License.

github-actions[bot] commented 2 months ago

Thanks for opening this pull request! :tada: Please consult the contributing guidelines for details on how to contribute to this project. If you need any assistance, don't hesitate to ping the relevant owner over Discord.

Topic	Owner
Wing SDK and standard library	`@chriscbr`
Wing Console	`@skyrpex`
Wing compiler and language design	`@chriscbr`
VSCode extension and language server	`@chriscbr`
Wing CLI	`@chriscbr`
Documentation	`@boyney123`
Examples	`@boyney123`
Wing Playground	`@skyrpex`

Chriscbr commented 2 months ago

It looks like we need to be smart about how we emit these reflection classes, otherwise we'll run into code generation issues. For example, supposing you have structs that refer to each other like these:

struct S1 {
  child: S2?;
}
struct S2 {
 child: S1?;
}

Then the naive way of generating meta-objects in JavaScript will look something like this:

new std.StructType("S1", {
  child: new std.OptionalType(
    new std.StructType("S2", {
      child: new std.OptionalType(
        new std.StructType("S1", <infinity>)
      ),
    }),
  ),
});

The example above is contrived, but in practice there are many recursive relationships between classes in Wing libraries and the SDK via method parameters and return types etc.

Chriscbr commented 2 months ago

The way I'm thinking of solving this is to "collect" the list of types referenced through helper functions in jsify.rs, assigning names to each of them, and then to emit somewhere around the beginning of preflight.js looking something like:

// prelude: we instantiate classes for all types, without their contents filled
let $s1_type = new std.StructType("S1");
let $s2_type = new std.StructType("S2");

// after: emit code to fill in type information, with cycles allowed
$s1_type.fields = { child: new std.OptionalType($s2_type) };
$s2_type.fields = { child: new std.OptionalType($s1_type) };

This code that defines reflection data for named types (classes, structs, interfaces, enums) can go into some kind of prelude section of preflight.js. Or even in a separate file. Then, wherever @type(MyStruct) is actually called, it will just emit a reference to the variable $s1_type.

Chriscbr commented 2 months ago

I've been making a little bit more progress. To do the approach above, we basically need to do some recursion on types. When we generate the javascript reflection code for S1, it needs to emit the reflection code for S2, and to emit that reflection code, we'd need to emit reflection code for S1. To avoid recursing indefinitely, we need to have a base case where if a type has already been explored (we've generated code for it), then we stop and reuse the existing result.

But we can't do that easily today because Type / TypeRef do not implement the Eq/Hash traits. Put another way, we have no way to tell if two types are equal. Part of the reason why it hasn't been easy to define equality on types is because we create duplicate type objects even for identical function signatures - there's no interning process going on. It's possible we can add a stub implementation of Eq/Hash that handles enough of the cases and works around this, but it might be worthwhile to see if we can change how we're storing types to avoid.

Chriscbr commented 2 months ago

Wow - this recursion problem really punches back!

So it seems like I fixed all of the issues with generating types that may have cyclic references in preflight. I've copied an example of a types.cjs file generated with my local build:

```js const std = require("@winglang/sdk").std; const $types = {}; $types.t1_interface_MyInterface = std.Type._ofInterface(new std.InterfaceType("MyInterface", "examples-valid.MyInterface")); $types.t2_interface_BaseInterface = std.Type._ofInterface(new std.InterfaceType("BaseInterface", "examples-valid.BaseInterface")); $types.t3_interface_IResource = std.Type._ofInterface(new std.InterfaceType("IResource", "@winglang/sdk.std.IResource")); $types.t4_interface_IConstruct = std.Type._ofInterface(new std.InterfaceType("IConstruct", "constructs.IConstruct")); $types.t5_interface_IDependable = std.Type._ofInterface(new std.InterfaceType("IDependable", "constructs.IDependable")); $types.t6_interface_IHostedLiftable = std.Type._ofInterface(new std.InterfaceType("IHostedLiftable", "@winglang/sdk.std.IHostedLiftable")); $types.t7_interface_ILiftable = std.Type._ofInterface(new std.InterfaceType("ILiftable", "@winglang/sdk.std.ILiftable")); $types.t8_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t9_interface_IInflightHost = std.Type._ofInterface(new std.InterfaceType("IInflightHost", "@winglang/sdk.std.IInflightHost")); $types.t10_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t11_str = std.Type._ofStr(); $types.t12_void = std.Type._ofVoid(); $types.t13_array_str = std.Type._ofArray(undefined, false); $types.t14_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t15_class_MyClass = std.Type._ofClass(new std.ClassType("MyClass", undefined)); $types.t16_class_Resource = std.Type._ofClass(new std.ClassType("Resource", "@winglang/sdk.std.Resource")); $types.t17_class_Construct = std.Type._ofClass(new std.ClassType("Construct", "constructs.Construct")); $types.t18_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t19_any = std.Type._ofAny(); $types.t20_bool = std.Type._ofBool(); $types.t21_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t22_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t23_array_str = std.Type._ofArray(undefined, false); $types.t24_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t25_array_str = std.Type._ofArray(undefined, false); $types.t26_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t27_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t28_fn = std.Type._ofFunction(new std.FunctionType(std.Phase.PREFLIGHT)); $types.t29_enum_MyEnum = std.Type._ofEnum(new std.EnumType("MyEnum", "examples-valid.MyEnum", {"VARIANT1": new std.EnumVariant("VARIANT1"), "VARIANT2": new std.EnumVariant("VARIANT2"), }));; $types.t30_struct_MyStruct = std.Type._ofStruct(new std.StructType("MyStruct", "examples-valid.MyStruct")); $types.t31_struct_Base1 = std.Type._ofStruct(new std.StructType("Base1", "examples-valid.Base1")); $types.t32_struct_Base2 = std.Type._ofStruct(new std.StructType("Base2", "examples-valid.Base2")); $types.t33_num = std.Type._ofNum(); $types.t34_array_opt_json = std.Type._ofArray(undefined, false); $types.t35_opt_json = std.Type._ofOptional(undefined); $types.t36_json = std.Type._ofJson(); $types.t1_interface_MyInterface.data.bases.push($types.t2_interface_BaseInterface.data); $types.t1_interface_MyInterface.data.methods["method1"] = new std.Method("method1", false, $types.t14_fn.data); $types.t2_interface_BaseInterface.data.bases.push($types.t3_interface_IResource.data); $types.t3_interface_IResource.data.bases.push($types.t4_interface_IConstruct.data); $types.t3_interface_IResource.data.bases.push($types.t6_interface_IHostedLiftable.data); $types.t3_interface_IResource.data.methods["onLift"] = new std.Method("onLift", false, $types.t8_fn.data); $types.t4_interface_IConstruct.data.bases.push($types.t5_interface_IDependable.data); $types.t6_interface_IHostedLiftable.data.bases.push($types.t7_interface_ILiftable.data); $types.t6_interface_IHostedLiftable.data.methods["onLift"] = new std.Method("onLift", false, $types.t8_fn.data); $types.t8_fn.data.params.push($types.t9_interface_IInflightHost.data); $types.t8_fn.data.params.push($types.t13_array_str.data); $types.t8_fn.data.returns = $types.t12_void; $types.t9_interface_IInflightHost.data.bases.push($types.t3_interface_IResource.data); $types.t9_interface_IInflightHost.data.methods["addEnvironment"] = new std.Method("addEnvironment", false, $types.t10_fn.data); $types.t10_fn.data.params.push($types.t11_str.data); $types.t10_fn.data.params.push($types.t11_str.data); $types.t10_fn.data.returns = $types.t12_void; $types.t13_array_str.data.child = $types.t11_str; $types.t14_fn.data.returns = $types.t12_void; $types.t15_class_MyClass.data.base = $types.t16_class_Resource.data; $types.t15_class_MyClass.data.interfaces.push($types.t1_interface_MyInterface.data); $types.t15_class_MyClass.data.properties["field1"] = new std.Property("field1", $types.t11_str); $types.t15_class_MyClass.data.methods["method1"] = new std.Method("method1", false, $types.t27_fn.data); $types.t15_class_MyClass.data.methods["method2"] = new std.Method("method2", true, $types.t28_fn.data); $types.t15_class_MyClass.data.methods["onLift"] = new std.Method("onLift", false, $types.t22_fn.data); $types.t15_class_MyClass.data.methods["onLiftType"] = new std.Method("onLiftType", true, $types.t24_fn.data); $types.t15_class_MyClass.data.methods["toInflight"] = new std.Method("toInflight", true, $types.t26_fn.data); $types.t15_class_MyClass.data.methods["isConstruct"] = new std.Method("isConstruct", true, $types.t18_fn.data); $types.t15_class_MyClass.data.methods["toString"] = new std.Method("toString", false, $types.t21_fn.data); $types.t16_class_Resource.data.base = $types.t17_class_Construct.data; $types.t16_class_Resource.data.interfaces.push($types.t3_interface_IResource.data); $types.t16_class_Resource.data.methods["onLift"] = new std.Method("onLift", false, $types.t22_fn.data); $types.t16_class_Resource.data.methods["onLiftType"] = new std.Method("onLiftType", true, $types.t24_fn.data); $types.t16_class_Resource.data.methods["toInflight"] = new std.Method("toInflight", true, $types.t26_fn.data); $types.t16_class_Resource.data.methods["isConstruct"] = new std.Method("isConstruct", true, $types.t18_fn.data); $types.t16_class_Resource.data.methods["toString"] = new std.Method("toString", false, $types.t21_fn.data); $types.t17_class_Construct.data.interfaces.push($types.t4_interface_IConstruct.data); $types.t17_class_Construct.data.methods["isConstruct"] = new std.Method("isConstruct", true, $types.t18_fn.data); $types.t17_class_Construct.data.methods["toString"] = new std.Method("toString", false, $types.t21_fn.data); $types.t18_fn.data.params.push($types.t19_any.data); $types.t18_fn.data.returns = $types.t20_bool; $types.t21_fn.data.returns = $types.t11_str; $types.t22_fn.data.params.push($types.t9_interface_IInflightHost.data); $types.t22_fn.data.params.push($types.t23_array_str.data); $types.t22_fn.data.returns = $types.t12_void; $types.t23_array_str.data.child = $types.t11_str; $types.t24_fn.data.params.push($types.t9_interface_IInflightHost.data); $types.t24_fn.data.params.push($types.t25_array_str.data); $types.t24_fn.data.returns = $types.t12_void; $types.t25_array_str.data.child = $types.t11_str; $types.t26_fn.data.params.push($types.t3_interface_IResource.data); $types.t26_fn.data.returns = $types.t11_str; $types.t27_fn.data.returns = $types.t12_void; $types.t28_fn.data.returns = $types.t12_void; $types.t30_struct_MyStruct.data.bases.push($types.t31_struct_Base1.data); $types.t30_struct_MyStruct.data.bases.push($types.t32_struct_Base2.data); $types.t30_struct_MyStruct.data.fields["base1"] = new std.Property("base1", $types.t20_bool); $types.t30_struct_MyStruct.data.fields["base2"] = new std.Property("base2", $types.t20_bool); $types.t30_struct_MyStruct.data.fields["field1"] = new std.Property("field1", $types.t33_num); $types.t30_struct_MyStruct.data.fields["field2"] = new std.Property("field2", $types.t34_array_opt_json); $types.t31_struct_Base1.data.fields["base1"] = new std.Property("base1", $types.t20_bool); $types.t32_struct_Base2.data.fields["base2"] = new std.Property("base2", $types.t20_bool); $types.t34_array_opt_json.data.child = $types.t35_opt_json; $types.t35_opt_json.data.child = $types.t36_json; module.exports = $types; ```

All of this works fine. We define the types first, then fill in the types' contents.

But it appears that separately, there's a problem with lifting std.Type instances from preflight to inflight. Namely, calling toInflight() sometimes blows the stack frame.

Let's look back at the example before:

struct S1 {
  child: S2?;
}

struct S2 {
 child: S1?;
}

And suppose we lift a type reflection:

let s1 = @type(S1);

inflight () => {
  log(s1.kind); // should print "struct"
}

In preflight, s1 is an instance of std.Type storing inside a std.StructType. To use it inflight, we have to generate some JS code that will get included in the inflight function, through toInflight(). What does the toInflight() implementation look like? The current naive implementation looks approximately like this:

  /** @internal */
  public _toInflight(): string {
    const args = [
      `"${this.name}"`,
      this.fqn ? `"${this.fqn}"` : "undefined",
      arrayToInflight(this.bases), // helper function that calls _toInflight() on each parent struct reflection
      mapToInflight(this.fields), // helper function that calls _toInflight() on each field reflection
    ];
    return `new std.StructType(${args.join(", ")})`;
  }

But as you can imagine, this means s1._toInflight() will end up calling s2._toInflight() which will then call s1._toInflight() and so on.

The solution isn't as obvious to me. One idea could be to do some kind of runtime tree traversal -- so we can find all unique types -- then define some variables for them, then call an altered method like toInflightWithContext() on each type so that it can refer to this map of variables (stopping the endless recursion) when it sees another type.

Another idea could be to totally revise how we're code generating these helper classes. But I'm not sure where I'd start with that to be honest.

eladb commented 2 months ago

Another idea could be to totally revise how we're code generating these helper classes. But I'm not sure where I'd start with that to be honest.

A sketch of a thought: instead of only generating types that are accessed, perhaps we could generate a single in-memory data structure that includes the entire app type system. Basically a big-ass JSON dump of the application's type info that the compiler holds in-memory. Then @type(x) will just access this at runtime (preflight-time).

So this:

class Foo {}

let x = @type(Foo);

Becomes this:

const $types = require("./types.json");

const x = std.Type._of($types["fqn.of.Foo"]);

MarkMcCulloh commented 2 months ago

instead of only generating types that are accessed, perhaps we could generate a single in-memory data structure that includes the entire app type system.

Depending on how this is done, this could be a huge performance hit with JSII. Unless the original jsii manifest could be reused somehow.

But it appears that separately, there's a problem with lifting std.Type instances from preflight to inflight. Namely, calling toInflight() sometimes blows the stack frame. But as you can imagine, this means s1._toInflight() will end up calling s2._toInflight() which will then call s1._toInflight() and so on.

@Chriscbr Maybe this is too far, but what if the std.Type API were changed so that all access to child type information is done through a preflight method instead of a property? iirc this is how the Type class in C# reflection works https://learn.microsoft.com/en-us/dotnet/api/system.type?view=net-8.0 (perhaps due to similar recursion problems?)

There's a hit to ergonomics, but I think it avoids the eager/cyclic lifting

struct S1 {
  child: S2?;
}

struct S2 {
 child: S1?;
}

let s1 = @type(S1);
let s2 = s1.getField("child");
let allFields = s1.getFields();

inflight () => {
  log(s1.kind); // should print "struct"
  log(s2.kind); // should print "struct"
}

monadabot commented 1 month ago

Console preview environment is available at https://wing-console-pr-7151.fly.dev :rocket:

Last Updated (UTC) 2024-09-27 20:40

monadabot commented 1 month ago

Benchmarks

Comparison to Baseline ⬜🟥⬜⬜⬜⬜⬜⬜⬜🟥⬜⬜⬜

| Benchmark | Before | After | Change | | :-- | --: | --: | --: | | version | 57ms±0.61 | 57ms±0.59 | +1ms (+1.11%)⬜ | | hello_world.test.w -t sim | 421ms±4.31 | 439ms±24.57 | +19ms (+4.4%)🟥 | | hello_world.test.w -t tf-aws | 1535ms±14.86 | 1519ms±9.52 | -16ms (-1.03%)⬜ | | jsii_small.test.w -t sim | 390ms±5.33 | 396ms±13.6 | +6ms (+1.47%)⬜ | | jsii_small.test.w -t tf-aws | 625ms±9.79 | 631ms±6.32 | +6ms (+1.02%)⬜ | | jsii_big.test.w -t sim | 3049ms±19.68 | 3075ms±22.1 | +25ms (+0.83%)⬜ | | jsii_big.test.w -t tf-aws | 3236ms±10.67 | 3256ms±22.84 | +21ms (+0.64%)⬜ | | empty.test.w -t sim | 382ms±4.28 | 374ms±5.36 | -7ms (-1.96%)⬜ | | empty.test.w -t tf-aws | 614ms±3.4 | 611ms±4.08 | -3ms (-0.54%)⬜ | | functions_10.test.w -t sim | 484ms±15.59 | 510ms±12.85 | +26ms (+5.47%)🟥 | | functions_10.test.w -t tf-aws | 2224ms±16.73 | 2234ms±12.46 | +10ms (+0.44%)⬜ | | functions_1.test.w -t sim | 415ms±5.03 | 414ms±3.33 | -2ms (-0.4%)⬜ | | functions_1.test.w -t tf-aws | 865ms±3.87 | 857ms±4.12 | -8ms (-0.91%)⬜ | ⬜ Within 1.5 standard deviations 🟩 Faster, Above 1.5 standard deviations 🟥 Slower, Above 1.5 standard deviations _Benchmarks may vary outside of normal expectations, especially when running in GitHub Actions CI._

Results

|name|mean|min|max|moe|sd| |----|----|----|----|----|----| |version|57ms|57ms|59ms|1ms|1ms| |hello_world.test.w -t sim|439ms|406ms|516ms|25ms|34ms| |hello_world.test.w -t tf-aws|1519ms|1502ms|1545ms|10ms|13ms| |jsii_small.test.w -t sim|396ms|386ms|448ms|14ms|19ms| |jsii_small.test.w -t tf-aws|631ms|617ms|646ms|6ms|9ms| |jsii_big.test.w -t sim|3075ms|3021ms|3117ms|22ms|31ms| |jsii_big.test.w -t tf-aws|3256ms|3220ms|3315ms|23ms|32ms| |empty.test.w -t sim|374ms|362ms|384ms|5ms|7ms| |empty.test.w -t tf-aws|611ms|602ms|619ms|4ms|6ms| |functions_10.test.w -t sim|510ms|467ms|532ms|13ms|18ms| |functions_10.test.w -t tf-aws|2234ms|2207ms|2271ms|12ms|17ms| |functions_1.test.w -t sim|414ms|408ms|425ms|3ms|5ms| |functions_1.test.w -t tf-aws|857ms|848ms|864ms|4ms|6ms|

Last Updated (UTC) 2024-09-27 20:45

mergify[bot] commented 1 month ago

Thanks for contributing, @Chriscbr! This PR will now be added to the merge queue, or immediately merged if rybickic/type-intrinsic is up-to-date with main and the queue is empty.

monadabot commented 1 month ago

Congrats! :rocket: This was released in Wing 0.85.18.

winglang / wing