Cannot reference a recursive type

[sdc395]

Hi

Using Typebox 0.32.31, it seems that a recursive type cannot be referenced. The code...

import { Type } from '@sinclair/typebox';
import { TypeCompiler } from '@sinclair/typebox/compiler';

const Node = Type.Recursive(Node => Type.Object({
    id: Type.String(),
    nodes: Type.Array(Node),
}), { $id: 'Node' });

const NodeRef = Type.Ref(Node);

TypeCompiler.Compile(NodeRef);

results in...

TypeDereferenceError: Unable to dereference schema with $id 'undefined'
    at Resolve (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/value/deref/deref.js:17:15)
    at Deref (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/value/deref/deref.js:24:11)
    at FromRef (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:380:42)
    at FromRef.next (<anonymous>)
    at Visit (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:514:31)
    at Visit.next (<anonymous>)
    at CreateFunction (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:571:32)
    at Build (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:585:30)
    at Code (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:615:16)
    at Object.Compile (/home/simon/typebox-issue/node_modules/@sinclair/typebox/build/cjs/compiler/compiler.js:620:31) {
  schema: { '$ref': 'Node', [Symbol(TypeBox.Kind)]: 'Ref' }
}

The actual schema I'm working with is more complex and, without using references, was causing Fastify to throw the following error (which I assumed was due to me reusing a sub-schema with an $id).

Failed building the serialization schema for GET: /complex-type, due to error reference "ComplexType" resolves to more than one schema

Perhaps this is a bug?

[sdc395]

I've just realised that the involvement of a recursive type was an assumption of mine. The following causes the same issue...

import { Type } from '@sinclair/typebox';
import { TypeCompiler } from '@sinclair/typebox/compiler';

const Vector = Type.Object({
    x: Type.Number(),
    y: Type.Number(),
}, { $id: 'Vector' });

const VectorRef = Type.Ref(Vector);

TypeCompiler.Compile(VectorRef);

Does TypeCompiler not support references?

[sdc395]

OK, so I had assumed that TypeCompiler had magical access to the referenced type. I'd somehow failed to notice the references optional argument to TypeCompiler.compile. Providing the Vector schema to TypeCompiler.Compile(VectorRef) fixes my error.

That said, there still seems to be an issue with recursive types.

import { Type } from '@sinclair/typebox';
import { TypeCompiler } from '@sinclair/typebox/compiler';

const Vector = Type.Object({
    x: Type.Number(),
    y: Type.Number(),
}, { $id: 'Vector' })

const VectorRef = Type.Ref(Vector)
const VectorDeref = Type.Deref(VectorRef, [Vector]);

// OK
TypeCompiler.Compile(VectorDeref);

const Node = Type.Recursive(Node => Type.Object({
    id: Type.String(),
    nodes: Type.Array(Node),
}, { $id: 'Node' }));

const NodeRef = Type.Ref(Node);
const NodeDeref = Type.Deref(NodeRef, [Node]);

// TypeDereferenceError: Unable to dereference schema with $id 'undefined'
TypeCompiler.Compile(NodeDeref);

[sinclairzx81]

@sdc395 Hi,

Please ignore this non-issue. I had assumed that TypeCompiler had magical access to the referenced type. My bad.

That's right, Ref types are string referenced aliases to some target type. To validate a type containing a reference type, you will need to inform the compiler about the target type so it can be dereferenced. There's a couple of way to achieve this. Consider for the provided Vector type.

const Vector = Type.Object({
    x: Type.Number(),
    y: Type.Number(),
}, { $id: 'Vector' })

const VectorRef = Type.Ref(Vector)

Option A: Pass references array to Compile

Most TypeBox functions have an overloaded function to pass a references array. The types passed on the array should all have $id identifiers such that the Ref can target it.

TypeCompiler.Compile(VectorRef, [Vector])

// similarly you can pass references to Value.Check

Value.Check(VectorRef, [Vector], { x: 1, y: 2 })

Option B: Defer target type

In 0.32.0, TypeBox added the Deref type which will return a de-normalized type. The following uses Deref to deference the target type meaning you don't need to pass the references array on Compile.

TypeCompiler.Compile(Type.Deref(VectorRef, [Vector]))

Auto Tracking References

In both cases above, you will need to carry around the reference + target types, however there are abstractions you can write over the top of Compile to auto track $id / $ref. The following is a quick implementation of this.

import { Type, TSchema } from '@sinclair/typebox'
import { TypeCompiler, TypeCheck } from '@sinclair/typebox/compiler'

// ------------------------------------------------------------------
// Auto Tracking References
// ------------------------------------------------------------------

const references: TSchema[] = []
let ordinal = 0

// wraps a type and ensures it has an $id. The type is added to the references array.
function Target<T extends TSchema>(schema: T): T {
  if('$id' in schema) {
    references.push(schema)
    return schema
  } else {
    const remapped = { $id: `type-${ordinal++}`, ...schema }
    references.push(remapped)
    return remapped
  }
}
// custom compile method
function Compile<T extends TSchema>(schema: T): TypeCheck<T> {
  return TypeCompiler.Compile(schema, references)
}

// ------------------------------------------------------------------
// Example
// ------------------------------------------------------------------

const Vector = Target(Type.Object({
  x: Type.Number(),
  y: Type.Number(),
}))

const VectorRef = Type.Ref(Vector)

Compile(VectorRef)

Hope this helps S

[sdc395]

Hi @sinclairzx81

Thank you for your response to my confusion. I'll read through it and try to get a grip. :-)

[sinclairzx81]

@sdc395 Hey

Will close off this issue as you will need to pass references arrays on various functions if using Ref. TypeBox will deference types if you supply the references array on various functions, however tracking references is something that must be handled by the implementer (as there's a few ways to approach it). You can use the "Auto Tracking References" above as a starter.

Cheers! S

[sdc395]

With regard to my "Vector versus Node" code above, I'm guessing the problem comes down to the way references are unpacked into duplicates of the referenced schema (an assumption based on the documentation of Type.Deref).

Would support for definitions solve this? Is there a workaround that would allow me to export a single schema containing references to recursive types from a package?

[sinclairzx81]

@sdc395 Hi,

Would support for definitions solve this? Is there a workaround that would allow me to export a single schema containing references to recursive types from a package?

Not necessarily, but let me explain.

Many versions ago, TypeBox included a Type.Namespace type that produced a $defs schematic. This type was removed however as the construction of $defs varied widely among json schema users, and anything TypeBox introduced was too narrow to meet ALL end user requirements. Many users were experiencing issues trying to port $defs schematics to Type.Namespace and finding out the referencing paths, and structures did not fit the type.

As such, a call was made to remove Type.Namespace in the short term which forced users to build out their own $defs and wrestle with Type.Ref, String Ref, and out of order definitions (things that TypeBox would have otherwise had to wrestle with internally). This decision was mostly prompted due to the uncertainty on how to appropriately handle $defs (with many of your questions here and recently noted on https://github.com/sinclairzx81/typebox/issues/882 being pressing questions internal to the library also)

All this said, I am planning to reintroduce Type.Namespace. As before, this Type would generate $defs schematics, however the type would be documented as being "extremely opinionated" and "specific to TypeBox". This following is the high level idea.

// Model generates referential $id based on property name
const Model = Type.Namespace({ 
  Name: Type.String(),
  Person: Type.Object({
    name: Type.Ref('Name') // reference the Name type
  })
})

// Infer This Way
type Model = Static<typeof Model>
type Name = Model['Name']
type Person = Model['Person']

// Get Schematics This Way
const Name = Type.Index(Model, 'Name')
const Person = Type.Index(Model, 'Person')

// Or this way
const Name = Model.$defs.Name
const Person = Model.$defs.Person 

Keep in mind, the above probably wouldn't solve the json-schema-to-typescript issue noted (as for the above to work, TypeBox is heavily reliant on internal tracked identifiers, and linear / flat structure of the $defs, and cannot predict in advance what exterior tools will generate). But there will be something to map into if you're prepared to follow TB conventions.

Hope this brings some insight Cheers S