Apply transformations to Dynamic References

tl;dr I would like to be able to apply Partial<T>, Omit<T, K> and other transformations to dynamic references that are resolved at runtime. If you are not familiar with TypeScript, Partial<T> is a utility type that makes all properties of T optional. Omit<T, K> is a utility type that removes properties K from T.

Use Case

TypeScript Example

Here's my use case; Basically I want to define generic interfaces that can be implemented to create a data source of a certain data type. Here's what I mean in TypeScript:

// definitions

type ReadInput<T> = { id: string }
type ReadOutput<T> = { item: T & { id: string; createdAt: string; updatedAt: string } }
interface Readable<T> {
  read(input: ReadInput<T>): Promise<ReadOutput<T>>
}

type CreateInput<T> = { data: T }
type CreateOutput<T> = { item: T & { id: string; createdAt: string; updatedAt: string } }
interface Createable<T> {
  create(input: CreateInput<T>): Promise<CreateOutput<T>>
}

type UpdateInput<T> = { id: string; data: Partial<T> }
type UpdateOutput<T> = { item: T & { id: string; createdAt: string; updatedAt: string } }
interface Updateable<T> {
  update(input: UpdateInput<T>): Promise<UpdateOutput<T>>
}

// implementation

type User = { name: string; email: string } // my data type

// my data source
class UserRepository implements Readable<User>, Createable<User>, Updateable<User> {
  private _users: Record<string, User & { createdAt: string; updatedAt: string }> = {}

  async read({ id }: ReadInput<User>) {
    return {
      item: { id, ...this._users[id] },
    }
  }

  async create({ data }: CreateInput<User>) {
    const id = Math.random().toString(36).slice(2)
    const user = { ...data, createdAt: new Date().toISOString(), updatedAt: new Date().toISOString() }
    this._users[id] = user
    return { item: { id, ...user } }
  }

  async update({ id, data }: UpdateInput<User>) {
    const user = this._users[id]
    if (!user) {
      throw new Error('User not found')
    }
    const updatedUser = { ...user, ...data, updatedAt: new Date().toISOString() }
    this._users[id] = updatedUser
    return { item: { id, ...updatedUser } }
  }
}

The thing is, my implementation is not actually a TypeScript class but a web server. Intead of functions/methods, the read, create and update operations are HTTP routes. I would like to be able to validate the input and output of each of these operation given a data schema for template argument T.

The Readable and Creatable Interfaces

Currently I can validate the IO of the Readable interface like this:

{
  "title": "ReadInput",
  "type": "object",
  "properties": {
    "id": { "type": "string" }
  },
  "required": ["id"]
}

{
  "title": "ReadOutput",
  "$defs": {
    "data": {
      "$dynamicAnchor": "T"
    }
  },
  "type": "object",
  "properties": {
    "item": {
      "allOf": [
        { "$dynamicRef": "#T" },
        {
          "type": "object",
          "properties": {
            "id": { "type": "string" },
            "createdAt": { "type": "string" },
            "updatedAt": { "type": "string" }
          },
          "required": ["id", "createdAt", "updatedAt"]
        }
      ]
    }
  }
}

And at runtime resolve the data schema for User:

{
  "$ref": "readable-output.json",
  "$defs": {
    "user": {
      "$dynamicAnchor": "T",
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "email": { "type": "string" }
      },
      "required": ["name", "email"]
    }
  }
}

(I can do the same thing for Creatable)

The Updateable Interface

The problem is with the Updateable interface, I would need a way of overriding the schema of T to set required: [] so that all properties are optional.

{
  "title": "UpdateInput",
  "type": "object",
  "$defs": {
    "data": {
      "$dynamicAnchor": "T"
    }
  },
  "properties": {
    "id": { "type": "string" },
    "data": {
      "allOf": [{ "$dynamicAnchor": "T" }, { "type": "object", "required": [] }] // this does not make any sense but you get the idea
    },
    "required": ["id", "data"]
  }
}

General Question

Is there any way to apply such transformations to dynamic references in JSON Schema? I'm specifically thinking about a way to make all properties of T optional or remove some properties from T like Partial<T> and Omit<T, K> in TypeScript.

Something like:

{
  "$defs": {
    "data": {
      "$dynamicAnchor": "T"
    }
  },
  "$override": [{ "$dynamicRef": "T" }, { "type": "object", "required": [] }]
}

{
  "$defs": {
    "data": {
      "$dynamicAnchor": "T"
    }
  },
  "$dynamicRef": "T",
  "$apply": [
    {
      "function": "partial"
    },
    {
      "function": "omit",
      "args": ["propertyA", "propertyB"]
    }
  ]
}

Thank you in advance for your time. I hope this is clear enough. If you have any questions, please let me know.

Frank

Generally, overriding behavior from references isn't a supported thing because JSON Schema is just a set of constraints. When you reference another (sub)schema, you're just appending those constraints to the local set.

For example

{
  "$id": "min-50",
  "type": "integer",
  "minimum": 50,
  "$ref": "two-digits"
}

{
  "$id": "two-digits",
  "minimum": 10,
  "maximum": 99
}

The $ref to two-digits adds the minimum: 10 and maximum: 99 constraints. The reference doesn't override the same-name constraints in the local schema. It just happens that the minimum is ineffectual due to the minimum: 50 present in the first schema. The result is accepting an integer in the range 50-99.

However, with dynamic references, you might be able to do something similar to an override.

{
  "$id": "min-50",
  "type": "integer",
  "$dynamicRef": "range"
  "$defs": {
    "base-range": {
      "$dynamicAnchor": "range",
      "minimum": 50
    }
  }
}

{
  "$id": "two-digits",
  "minimum": 10,
  "maximum": 99
}

{
  "$id": "override",
  "$ref": "min-50",
  "$defs": {
    "new-value": {
      "$dynamicAnchor": "range",
      "$ref": "two-digits"
    }
  }
}

If you start evaluation at min-50, then you'll validate an integer in the range 50-inf. The dynamic reference resolves to the local range anchor and adds a minimum: 50 constraint.

However, if you start evaluation at override, then the dynamic reference resolves to the range achor in override instead. This means that the minimum: 50 constraint isn't ever added, and you effectively override it with the constraints in two-digits. This validates the full 10-99 range.

(I think this may be the missing piece to supporting polymorphism! I'll continue to work on this, and I'll write a blog post if I come up with anything.)

There is no way in JSON Schema to apply a schema and remove a constraint. That means you can't do something like Omit or Partial. You have to change the way you approach the problem to thinking about building schemas up rather than breaking them down.

Here's how I would implement the schemas for the update types.

{
  "$id": "https://json-schema.hyperjump.io/schema",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$ref": "update-output",

  "$defs": {
    "T": {
      "$dynamicAnchor": "T",
      "$ref": "#/$defs/Partial-T",
      "required": ["foo"]
    },
    "Partial-T": {
      "$dynamicAnchor": "Partial-T",
      "type": "object",
      "properties": {
        "foo": { "type": "string" }
      }
    },

    "update-input": {
      "$id": "update-input",

      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "data": { "$dynamicRef": "#Partial-T" }
      },
      "required": ["id", "data"],

      "$defs": {
        "data": { "$dynamicAnchor": "Partial-T" }
      }
    },
    "update-output": {
      "$id": "update-output",

      "type": "object",
      "properties": {
        "item": {
          "$dynamicRef": "#T",
          "$ref": "output"
        }
      },
      "required": ["item"],

      "$defs": {
        "data": { "$dynamicAnchor": "T" }
      }
    },
    "output": {
      "$id": "output",

      "properties": {
        "id": { "type": "string" },
        "createdAt": { "type": "string" },
        "updatedAt": { "type": "string" }
      },
      "required": ["id", "createdAt", "updatedAt"]
    }
  }
}

Hey, @gregsdennis and @jdesrosiers !

Many thanks for such quick replies. Your answers make a lot of sense:

Generally, overriding behavior from references isn't a supported thing because JSON Schema is just a set of constraints. When you reference another (sub)schema, you're just appending those constraints to the local set.

There is no way in JSON Schema to apply a schema and remove a constraint. That means you can't do something like Omit or Partial. You have to change the way you approach the problem to thinking about building schemas up rather than breaking them down.

Unfortunately, the system I'm currently building can only require the user to provide a single data schema, and I have to work with that. I can't ask the user to provide both a partial and full schema.

What I will probably end up doing is shipping my feature without the partial update and eventually implement my own dereferencing function to handle URIs with special query parameters like { $dynamicRef: "#T?partial=true&omit=field1,field2" }. This way, I can dynamically modify references.

Many thanks again for your time and help! I really appreciate it.

Please feel free to close this issue.

Best regards,

Frank

What I will probably end up doing is shipping my feature without the partial update and eventually implement my own dereferencing function to handle URIs with special query parameters like { $dynamicRef: "#T?partial=true&omit=field1,field2" }. This way, I can dynamically modify references.

If your tool is going to be publicly available, please be sure that you document (advertise?) this deviation from spec behavior. Ideally, you'd support the spec behavior and then hide this special support behind a config option so the user can opt in.

@gregsdennis

Sorry for the late reply;

The usage of JSON schema is actually hidden in our product. From the user's perspective, schemas are built using our own schema-builder library called Zui (forked from Zod).

Even if it's internal only, I'm still trying to be as close as possible to the spec so that it's easier to integrate with other tools and collaborate with other teams. For this purpose, I will make sure to thoroughly document this deviation from the spec behavior.

Many thanks again for your time and help,

Frank

json-schema-org / json-schema-spec