Typescript type inference from mongoose Schema

nitzanhen commented 3 years ago

Do you want to request a feature or report a bug? feature

What is the current behavior? Defining a model in a project with mongoose & typescript currently involves defining the schema twice - once as a mongoose schema, and once as a Typescript interface/type. This makes maintenance more difficult, especially for large models or models with nested fields/schemas.

What is the expected behavior? I think it would be great (both in terms of convenience and of good coding) if there was an official way to infer the model as a Typescript type from the schema. I actually have created a prototype implementation in one of my projects:

type Unboxed<T> = T extends String
  ? string
  : T extends Number
  ? number
  : T extends Boolean
  ? boolean
  : T;

type MongooseModel<Schema extends mongoose.SchemaDefinition> = {
  [K in keyof Schema]: Schema[K] extends (infer T)[] //Array
    ? T extends SchemaDefinition
      ? MongooseModel<T>[] //Nested schema
      : MappedField<T>[]
    : Schema[K] extends { type: MapConstructor; of: infer V } //Map type
    ? Record<string | number, V>
    : Schema[K] extends SchemaDefinition //Nested schema
    ? MongooseModel<Schema[K]>
    : MappedField<Schema[K]>;
};

type MappedField<F> = F extends { type: infer T }
  ? T extends new (...args: any) => infer S
    ? Unboxed<S>
    : T
  : F extends new (...args: any) => infer T
  ? Unboxed<T>
  : boolean;

However, there are two main things to be noted about this: First of all, this prototype something I put a few hours into, and supports all schema features I had in the project I built this for. Therefore, it should support Mongoose's schemas' basic features, but probably not all of them. That being said, I think most of the information that would be declared in a Typescript interface (e.g. the type of a field or its existence; some information, such as validation, default values or uniqueness, cannot be declared in Typescript unfortunately) can be deduces from the schema using Typescript's tools.

Second, and more important, is that this was built from outside, looking at mongoose as a black box of sorts. If we implement something of the sort inside mongoose, there may be additional types exposed to us that can make this definition easier or clearer.

as for the API, I think something akin to

  type Person = mongoose.infer<typeof personSchema>

(this draws inspiration from Zod's api) would be good, although it depends on the way this feature is eventually implemented, of course.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version. Node.js v14.15.1, mongoose v5.11.8. I'm using typescript 4.1.2; the implementation would need to use Typescript's infer keyword heavily, so the minimal version of Typescript that mongoose can be used with would need to support it.

nitzanhen commented 3 years ago

Also, the prototype I referenced above can probably be generally improved; any and all suggestions welcome.

I'll create a small repo which showcases this feature in a minimal mongoose-typescript setup when I can.

jcamden commented 3 years ago

I've been using ts-mongoose for this. It's somewhat limited since the flow is necessarily: create actual JS schema -> create typings -> provide them to TS. That means it lacks some of Typescripts utility types, but still pretty useful. FWIW, I agree with you that, now that mongoose is moving towards supporting Typescript, it would make sense to incorporate something like ts-mongoose natively 👍

vkarpov15 commented 3 years ago

We have a few ideas about how to support this, but have not implemented this yet. We will likely add this in the future.

nitzanhen commented 3 years ago

Ok, Great! Not to be too intrusive, but could you please elaborate on your ideas?

I'd love to help implement this if I can.

vkarpov15 commented 3 years ago

Nothing concrete enough to provide a code sample yet. But I'm very much open to suggestions @NitzanHen

nitzanhen commented 3 years ago

Ok, so I've done a bit of research. As I see things now, my original method is the only reasonable to go about this - use Typescript's infer keyword to define a higher-order type which maps a schema definition (to be exact, its type) to an "entity" type, which would be used elsewhere around the app. I'll name this higher-order type Entity (In my original post, it was MongooseModel), but do suggest a better name if you have one.

Incorporating `Entity` into Mongoose

First, let's assume we have a working Entity type, and talk about how to incorporate it into Mongoose.

Given a type Def which represents a schema definition, Entity works by mapping each of its fields into a "non-schema" field (e.g. String or { type: String } to string). For that to work, we need to extract, from a given schema definition, its explicit type. For example, in the schema definition:

const personDefinition = {
   name: String.
   nickname: { type: String, required: false } 
}

to extract a type Person out of this definition using Entity, we would need to treat personDefinition as an object of type { name: String, nickname: { type: String, required: false } }, as opposed to treating it simply as a SchemaDefinition.

In the app I mentioned in my original post, I achieved this by using Typescript's as const assertion - which tells TS to infer the object's type in the most precise way possible (as well as making it read-only - which would make sense for a "const" object, but is irrelevant here). I had done something similar to:

const personDefinition = {
   name: String.
   nickname: { type: String, required: false } 
} as const;

type Person = Entity<typeof personDefinition>

However, since we're able to declare the types of Mongoose's Schema itself, we can achieve the same result without changing the actual "flow" of defining a schema. We can use a very convenient pattern involving a generic constraint; generally, it works by replacing the type of an argument in a function or class with a generic which extends it. Doing this allows typescript to treat the type in the most precise way (explicitly, as we indeed want TS to see the schema definition in our case), while still constraining the end-user that calls the constructor into passing a valid value for that argument (in our case - a valid SchemaDefinition).

Let's start by seeing the code I'm proposing, then I'll (lengthily) explain :

//index.d.ts
//Somewhere in the file
type Entity<Def extends SchemaDefinition> = { /* implementation... */ }

//...
//Schema declaration, line ~1350
class Schema<
    Definition extends SchemaDefinition = SchemaDefinition, //First added generic
    E extends Entity<Definition> = Entity<Definition>, //Second added generic
    DocType extends E & Document = E & Document,
    M extends Model<DocType> = Model<DocType>
  > extends events.EventEmitter {
 /* ... */ 
}

//...
//Somewhere else in the file
/** Extracts the Entity of a given schema. */
export type infer<S extends Schema> = S extends Schema<any, infer E, any, any> ? E : never

//user-land
const personSchema = new Schema({ /* ... */ })

type Person = mongoose.infer<typeof personSchema>;

const PersonModel = mongoose.model('Person', personSchema);
//Use Person, PersonModel...

As you probably know, currently (looking at #9725), Schema receives between one and two generics - one for the document type (DocType) and one for the model type (M). This matches the model function's generics. By adding another generic to Schema, which Typescript will know to fill in itself, we can achieve elegant and convenient way to infer entities from the schema definition.

My proposition is basically this: add two generics to Schema, such that in total it receives four generics, all of which have a default value. The first generic would be used for typing the definition explicitly, and would be passed to the second generic - which would be- the Entity type derived from it (which I honestly am not sure is absolutely necessary, but would at the very least improve readability). The third and fourth would be the current generics, for the document type DocType and model type M. However, the document type should probably be modified too - instead of extending and defaulting to Document, it should extends and default to E & Document, where E is the second generic (the entity type) - that would, by the way, improve Schema typing of documents associated with it app-wide.

One downside to this modified typing is that, because of Typescript's constraints, the definition schema type and entity type have to come first (generics with defaults depending on another generic cannot be defined before that generic, apparently). This can be a minor inconvenience, because if a user would want to explicitly specify the document and model types (DocType and M), they would have to specify all four generics, which defeats the point of using the explicit Definition type in the first place (ideally, we could use partial generic inference to solve this issue, but TS doesn't have that, and won't be having that anytime soon). However, this downside has a simple workaround - the model function has only two generics as I mentioned earlier, and there's no reason for this to change as far as I can see (the added generics to schema are relevant to its declaration, not to the model itself). Therefore, a user that would need to override the TS's automatically inferred types (e.g. to add statics or methods) could do that through the model.

I believe this method provides a clean and easy way to infer TS types from the mongoose schema.

nitzanhen commented 3 years ago

Implementing Entity

As for implementing the Entity type, I believe we can simply improve on the MongooseModel type I provided in my original post, providing it with the logic to handle all the different types that Mongoose supports.

Below is a list of all the types & features of Mongoose schemas that Entity would need to consider. Please review it; suggestions are always welcome, and I want to be sure I didn't miss anything:

String - mapped to string
Number - mapped to number
Date - mapped to Date
Buffer - mapped to (Node.js) Buffer
Boolean - mapped to boolean
Mixed - mapped to any
ObjectId - mapped to an ObjectId instance (see note below)
Array - mapped to an array, taking the element type into consideration.
Decimal128 - mapped to number
Map - mapped to MongooseMap (taking the of parameter as the value of the record); should this be (perhaps optionally) be mapped to a Record instead?
Schema - mapped recursively (see the original MongooseModel for the gist of it)
A field with required: false should be mapped to an optional field.
A String fields with an enum parameter should be mapped to a string union instead of simply string.

Note: We need to consider maybe implementing multiple, similar variants of Entity - for example, for documents with populated/de-populated fields. I'm still thinking about the right approach to this need, and any insights or suggestions are welcome.

Have I missed something? It seems that the changes I'm suggesting (both for implementing Entity and incorporating it) shouldn't be a large amount of code overall, so I'll start working on everything on a fork of the repository, but I'm awaiting your feedback of course.

BorntraegerMarc commented 3 years ago

It would be great if this feature would support Schema.loadClass as well. So methods are typesafe and can be only defined once:

export const BlaSchema: Schema = new Schema({
    displayName: {
        type: String,
    }
});

export class Bla {
    speak(): void {
        console.log(this.displayName); // <-- Currently this will fail to transpile because it doesn't know "displayName" should be part of the class "Bla"
    }
}

BotSchema.loadClass(Bla);

nitzanhen commented 3 years ago

@BorntraegerMarc To be honest, I didn't know about Schema's loadClass method! it's quite neat!

I spot two problematic points in implementing type safety for loadClass (and the "flow" associated with it, which your code is a good example for) - the first is, as you've pointed out, that the class to be loaded (Bla in your example) has no knowledge of the fields defined on the schema it is purposed to be loaded to. The second is getting Typescript to acknowledge the loaded methods, statics and virtuals after having called loadSchema.

The latter, as far as I know, has no "proper" solution in Typescript - you would have to explicitly assert, somewhere, that BlaSchema now has new fields (methods, statics and virtuals). That is, since the type of a class object (or any variable, for that matter) can't be modified by invoking one of its methods. This may become an obstacle, since - if I got everything right - instance methods need to go on the Entity-Document combination type (that the model deals with), while static methods need to go on the Model class itself. Maybe I'm missing something.

As for the former, I think it should be pretty easily doable using Typescript's declaration merging feature - the class remains as it is, and beside it an interface with the same name (e.g. Bla) is declared. Typescript merges the fields of both types - meaning if you declare an interface that's identical to the type inferred from the schema (using Entity described in my proposition), e.g. Bla from BlaSchema, and it carries the same name as the class to be loaded (as is the case here), Typescript will know about the interface's fields in the class methods (and, of course, will still know to point out which fields don't on the schema, or which ones have been misused - preserving type safety). In practice, not much code needs to be added:

//Note: the `Schema` type annotation in your example needs to be removed for this to work!
export const BlaSchema = new Schema({ 
    displayName: {
        type: String,
    }
});

export interface Bla extends mongoose.infer<typeof BlaSchema> {}; //New line; merges with the class
export class Bla {
    speak(): void {
        console.log(this.displayName); // Now typescript should know about this
    }
}

//For convenience
export interface BlaDocument extends Bla, Document {}

BotSchema.loadClass(Bla);

//...
//Later, when declaring the model, we need to explicitly declare the new methods from `Bla`.
/** @todo: test this; do the static and instance methods of Bla need to be separated? */
const BlaModel = mongoose.model<BlaDocument, Model<BlaDocument & Bla /*??*/>>("Bla", BlaSchema)

My main assumption here is the the inferred type, e.g. Bla, can be "stuffed" into an interface (Entity<> returns a type, not an interface; it may be still be possible to create define Bla as an interface as I've done above, since theoretically Bla should be well known to Typescript, but this needs to be validated).

I haven't played around with loadClasss yet, and will do so when I have time. However, I think your suggestion is shaping up to be a feature of its own (depending on what I'm suggesting here), and I think it accordingly deserves a card of its own. Could you open one for it? I'd be glad to contribute with what I've explained above, as well as explain what method I'd used in my projects to define and declare methods, statics and virtuals (since it's quite easy to fall into inelegant or boilerplate-full code).

BorntraegerMarc commented 3 years ago

@NitzanHen I see your point: I also wouldn't know of a way for the problem nr. 2: "getting Typescript to acknowledge the loaded methods".

So about the general loadClass typesafness problem: I guess the only thing to do (as you already explained) would be to just document the "official" mongoose way how to declare loaded methods, statics and virtuals after having called loadClass. Wouldn't you agree it's more of a documentation problem and less a feature since it's not possible to do in typescript? Especially since I've seen a lot of different approaches to what we're trying to do here.

If so, I'm happy to open a new issue with some documentation suggestions.

Does anyone disagree with @NitzanHen's approach that it is the best / cleanest one?

On a different note: When copy/pasting your code typescript complains Namespace '"mongoose"' has no exported member 'infer'.. Would you mind explaining what you meant with mongoose.infer?

Furthermore: according to my tests extends mongoose.infer<typeof BlaSchema> is not really needed. E.g:

export const BlaSchema = new Schema({
    displayName: {
        type: String,
    },
});

export interface Bla {
    displayName: string;
}

export class Bla {
    speak(): void {
        console.log(this.displayName);
    }
}

export interface BlaDocument extends Bla, Document {}

BlaSchema.loadClass(Bla);

const BlaModel = mongoose.model<BlaDocument>('Bla', BlaSchema);

BlaModel.create({ displayName: '' }).then((b) => {
    b.speak();
});

EDIT: I'm an idiot 😄 Just noticed now that mongoose.infer is your proposal from this issue 😄

BorntraegerMarc commented 3 years ago

One more thing that came to my mind when dealing with type inference from mongoose Schema: defaults should be respected from the Schema. E.g:

import { Document, model, Schema, SchemaTypes } from 'mongoose';

export const BlaSchema = new Schema({
    displayName: {
        type: SchemaTypes.String,
    },
    language: {
        type: SchemaTypes.String,
        required: [true],
        default: 'english',
    },
});

export interface Bla {
    displayName: string;
    language: string;
}

export interface BlaDocument extends Bla, Document {}

const BlaModel = model<BlaDocument>('Bla', BlaSchema);

BlaModel.create({ displayName: '' }); // <--- This line throws TS error "No overload matches this call. Property 'language' is missing."

hgrubst commented 3 years ago

It would be great if this feature would support Schema.loadClass as well. So methods are typesafe and can be only defined once:

@BorntraegerMarc

I had the same issue and managed to overcome in the following way :

export class Bla { }  // <--- empty class that will be extended in the loadClass underneath. Typescript also merges this with the Bla interface giving you access to all fields
export interface Bla extends mongoose.Document {   // <----- typescript interface defining the properties for Bla
    displayName: string
    speak(): void;
}

export const BlaSchema: mongoose.Schema = new mongoose.Schema({
    displayName: {
        type: String,
    }
});

BlaSchema.loadClass(class extends Bla {  //<---- anonymous class that extends Bla so is now aware of the props of Bla
    speak(): void {
        console.log(this.displayName);
    }
});

Its not perfect as you still have the original problem that this issue describes eg duplication of mongoose and typescript definition. It would be neat if we could just define the whole schema as a class (and not just virtuals and methods) and use loadClass from there.

nitzanhen commented 3 years ago

@UncleVic I'm not sure what you're referring to. This thread discussed a suggestion related to inferring typescript types from a Mongoose schema. If it's related, please clarify, otherwise - please open another issue and describe your problem thoroughly.

nitzanhen commented 3 years ago

@BorntraegerMarc Wouldn't you agree it's more of a documentation problem and less a feature since it's not possible to do in typescript? Especially since I've seen a lot of different approaches to what we're trying to do here.

I totally agree. As far as I know Mongoose doesn't have proper documentation for Typescript users yet, and having implemented official Typescript types, now would be a good time to start working on them in my opinion. That being said, it's a whole lot of work, and I'm guessing most maintainers of this library have higher priorities.

Have you opened another card for it?

Also, sorry for disappearing, I'm having a busy couple of months. I'll try to make some progress with this when I'm able to. And of course, more suggestions or contributions are always welcome.

animaonline commented 3 years ago

Here's how I solved this

const UserSchemaFields: Record<keyof IFFUser, SchemaTypeOpts<any> | Schema | SchemaType> = {
    username: {
        type: String,
        unique: true
    },
    email: {
        type: String,
        unique: true,
    }
}
const UserSchema = new Schema(UserSchemaFields)

Now there's no need to double define interfaces.

dan-cooke commented 3 years ago

To follow on from @animaonline

Heres a TypedSchema function that will enforce strict typing on your schema definition based on a TS interface.

Unfortunately it will still require you to maintain the SDL and the TS Interface, so a change to the underlying TS interface will need to be made in the Schema definition as well, which is a little annoying.

But to work backwards and infer the type from the Schema Definition would require internal changes to mongoose I believe.

Developed with mongoose 5.12.0

export function TypedSchema<ISchema>(
  schema: SchemaDefinition<_AllowStringsForIds<LeanDocument<ISchema>>>
) {
  return new Schema(schema);
}

Usage

export interface ISymbol {
  symbol: string;
  securityName: string;
  isEtf: boolean;
  exchange: string;
  country: string;
  quote: ObjectId;
}
export const Symbol = TypedSchema<ISymbol>({
  symbol: String,
  securityName: String,
  isEtf: false,
  exchange: String,
  country: String,
  quote: { type: Schema.Types.ObjectId, ref: `Quote` },
});

nitzanhen commented 2 years ago

Seems this feature was implemented in #11563. Thanks for everyone involved!

Automattic / mongoose