domoritz commented 5 years ago

This is a discussion about the future of Typescript JSON schema generators.

TL;DR

Use https://github.com/vega/ts-json-schema-generator/ if you want a robust but opinionated schema generator. It has good support for aliases and it continuously updated. This is probably the right option for most people.
Use https://github.com/YousefED/typescript-json-schema if you want a less robust but hackable schema generator.

domoritz commented 4 years ago

@nonara Do you have a timeline for when I can take your tool for a spin? I have a case this generator does not support right now and I don't want to spend more time on it if there will be an alternative soon.

export const Foo = {
    x: "x",
    y: "y",
};

export type MyType = keyof typeof Foo;

nonara commented 4 years ago

@domoritz Ah, yeah. I can imagine covering all typeof would require essentially re-creating typescript's parser! Definitely not sustainable!

I need to wrap this up anyway, as I've got to get back to for-profit work next month. I'm pushing to get it done in the next 2-3 weeks. Which should definitely be achievable! There's not too much left to do, and I'll be devoting even more time to ensure it's ready on target.

What I'll do is make it public at the end of the month (at the latest) regardless of where it's at. Though I don't anticipate it not being ready to use. At that point, I may need some help filling out unit testing and improving coverage. If you and any others would like to get involved, we can work out any kinks or options changes before graduating to a full release.

My hope is that it will serve as a foundation for typescript-type / json translation, and that in moving forward, we can collectively improve and maintain it as a community effort!

HoldYourWaffle commented 4 years ago

That's so great to hear!

I'm currently in the process of starting up a new project which could massively benefit from this, and I'd love to take it for a spin in the next few weeks and help work out any kinks if necessary.

domoritz commented 4 years ago

@nonara Sounds good. Do you think it would make sense to put a work in progress version online so that you can get feedback on it?

Also, would you be interested in having us host the project in the Vega org for added visibility? I could create a repo and give you admin access. If you want to do that, what is a good name for the repo?

HoldYourWaffle commented 4 years ago

Speaking of visibility, is there a way to point people that stumble upon the "legacy" modules to the "new" one?

domoritz commented 4 years ago

Which ones, in particular, are you talking about? I put a pinned issue in the one I have been working on.

nonara commented 4 years ago

I'm currently in the process of starting up a new project which could massively benefit from this, and I'd love to take it for a spin in the next few weeks and help work out any kinks if necessary.

@HoldYourWaffle Great! Will be grateful for the help!

@nonara Sounds good. Do you think it would make sense to put a work in progress version online so that you can get feedback on it?

@domoritz It definitely will. The reason that I haven't yet is because I believe that it would have confused and frustrated people trying to keep up. The whole project structure and each of its parts have been completely rebuilt several times with significant differences. The overall architecture was extremely fluid as I was diving in to the typescript code-base as well as imagining or encountering new crazy scenarios which demanded entirely re-designing the approach, several times over.

That was the reason for this taking so much longer. I don't imagine that we're at the end of discovering new quirks and oddities, but I do believe that we're nearly at a point where I can say it's not going to be shifting around much.

And to be honest, I'm probably looking forward to having more minds involved on it than anyone else! I'm confident in what's been done, but I'm more excited to see where it can go and how it can be improved as the bright minds behind the existing work such as ts-json-schema-generator and the others join in.

Also, would you be interested in having us host the project in the Vega org for added visibility? I could create a repo and give you admin access.

Thanks! NPM-wise, it will need to be released under its own org, because it's modular. With that in mind, I think it'd be good to use the same org-name on GitHub, but I'd be glad to add you as admin, as I know you've been championing this over several projects for some time!

nonara commented 4 years ago

@HoldYourWaffle Assuming everyone is happy and we've decided to move forward with the new one, I believe GitHub allows some forms of deprecating / forwarding. Am I right on that?

Worst case, we can always put notices up.

nonara commented 4 years ago

Here's a look at the package structure for the new org @type-schema:

json-schema (Full typed JSON Schema drafts - intended to serve the type-schema suite as well as any other project which needs up-to-date JSON Drafts and/or Ts-Extras draft)
core - Core package for converting types to JSON schema (By itself, can serve for programmatic use)
plugin - Combined with core, it allows TSC to emit schema files, and also includes the language service component for autocompletion and validation of tags
validation - Runtime type validation addon (uses AJV)
cli - Allows CLI use
setup - Quick NPX / DLX tool to install the packages needed & setup environment (has CLI prompt / flags)

Validation, CLI, and Setup are at a lower priority, and may not be entirely finished before the rest goes live.

Expect core visibility in the next few weeks

Edit: In case anyone is wondering, the reason for modularity is mainly to keep it light-weight. Helpful as they are, the package requirements for inquirer, etc are bloated, and probably most will never need the CLI, especially if using the plugin.

Meanwhile

If anyone wants to have a look, I'd love feedback or thoughts on the structures in json-schema, which is now public.

https://github.com/type-schema/json-schema

HoldYourWaffle commented 4 years ago

I'll be sure to take a look at it once my project is at that stage!

Which ones, in particular, are you talking about?

I honestly have no idea anymore. It's been months since I last looked into this, but I remember starting this whole endeavor because there were like 3 or 4 forks that all seemed to do things just a little differently.

as well as imagining or encountering new crazy scenarios which demanded entirely re-designing the approach, several times over. That was the reason for this taking so much longer.

I feel like I relate to this way too much.

the bright minds behind the existing work such as ts-json-schema-generator and the others join in

no pressure

@HoldYourWaffle Assuming everyone is happy and we've decided to move forward with the new one, I believe GitHub allows some forms of deprecating / forwarding. Am I right on that?

I think they do, but I've never really looked into it that much. I know you can archive projects, but that's something only the maintainers can do.

Combined with core, it allows TSC to emit schema files, and also includes the language service component for autocompletion and validation of tags

This just made my ~day~week.

In case anyone is wondering, the reason for modularity is mainly to keep it light-weight.

I couldn't agree more.

timini commented 4 years ago

Watching with baited breath..

Have you guys see https://github.com/google/intermock also?

domoritz commented 4 years ago

Btw, I couldn’t wait any longer and added support for object and array literal expressions to this library. There are a few new test cases as well.

nonara commented 4 years ago

Hi folks! Got an update for you, and some code, as promised.

Update

Perhaps surprisingly, the most challenging part of this whole endeavour was the options and tags system. Because this is a full-scale compiler with various usages, the options and tags have a lot of metadata attached to each. To name a few, each option has groups, targets, a category, and flags. These define the contexts to which they apply, how they're implemented, whether they cascade to child nodes, etc.

Using this metadata, we programmatically filter and create our option sets for each context. We also need to replicate this approach via the Type system to derive Types for various option sets. This is required for some aspects of the compiler to work, such as auto-completion and the AST nodes' option properties.

I had this completed, but as I was wrapping things up, the mapped types for the sets dragged TS down to a crawl both during compile and LS computation integration in the IDE. It was so bad that I could type about 5 keystrokes before a 5 second pause. You can imagine the fun that added!

This unfortunately cost me about a week of doing what I could to refactor in hopes of optimizing or coming up with another strategy. While optimizing helped some, it was still too slow. Ultimately, it meant I had to build a portion of the build system, early, although I'd planned it for after the initial release. That took another 10 days or so.

With all that said - the good news is - all of that is finished and it works great.

What's left?

Very little. At this point, it's mainly just a few last bits in the parser to finish integrating the new TsExtras additions. Everything on the AST and render side is already done for those. I also need to implement rules and readonly.

Overall, it should likely only take a few days to wrap the final bits into the parser.

Finally, we'll sync up the latest set of tests from @domoritz and get everything passing. If there are any discrepancies beyond a simple code bug, I'll shelve those and address after I post the source.

What can I see?

I know everyone is anxious to see it. I'm excited to take it live! But I also didn't come this far to jump the gun, now. Thank you all for your patience! When I started, I didn't realize it would be a compiler! That realization has made me feel a little better about the time taken and abundance of caution put into its structure and naming conventions.

What I'm hoping is that we can collectively work out the rest together before we make an official RC. Discussions need to be had about versioning, etc. But we'll get there soon!

Source Outline

In the mean time, here is an outline package I've put together so you can see some of the key source & definition files to help you get a handle on it before the full monty is released:

StackBlitz: https://stackblitz.com/edit/type-schema-outline GitHub: https://github.com/type-schema/outline

Notes:

This is not a compilable project, so it's sufficient to just check it out on StackBlitz.
Unfortunately linebreaks and comment headers are stripped from the AST *.d.t.s files. I like code that reads nicely, but this should be fine for now.

How to look at it

Start in /main.ts and look at generateSchema() to see the steps for compilation along with each hook
Have a look at the SchemaNode structures that can be manipulated during hooks in /ast/nodes and /ast/_schema-node.ts for the base (also see utilities in /ast/utilities)
Have a look at /options/collections/base.collection.ts to see what the main options structure will look like. Venture into the other *.collection.ts to see the option sub-categories
Check out OptionRule in /options/_options.ts for a glimpse at how the rules option is implemented. (Rules will likely be used more often than base hooks)
Take a look at /generated/option-types.ts to see the generated option sets with documentation for all their various contexts

What's next?

Hopefully this will help give a bit of a picture of what's going on! There's a lot more going on under the hood, but I think this is a good start. I don't like that the AST d.ts files aren't nicely formatted, but you get the gist.

I'll post a status update next weekend, at which point I should be wrapping up or done with Dominik's tests.

Beyond this point, I'm hoping the community will help in building out unit tests and in join in discussion regarding structure, public API, options, performance etc. After it's up and going, I'll be transitioning to a 2-3day per week schedule for the compiler, so if folks want to help filling out the tests, I can devote time toward finishing the cool frills like LS extension (autocomplete) more quickly!

Until then -- cheers!

nonara commented 4 years ago

Hi all. Here's the latest...

Update

As I was working on integrating Functions into the Parser, I came across a fairly big issue.

There was no support for declaration merging!

To my knowledge, the other libraries do not have this either. I considered deferring support, however, for a number of reasons, including that function overloading relies on it and I was already going to be refactoring both the AST and the parser, I realized that it would be best to add it now.

Essentially, it means that we pull the members from Symbol, as opposed to the Type. Formerly, the parser primarily walked the full Type structure, using Symbol as needed in some special cases. The new logic will prefer Symbol where possible. (Some things do not have Symbol such as primitives and anonymous objects).

As of today, this is all fully integrated aside from the changes in the Parser. On the parser side, it's not a tremendous change. Nearly all primary logic remains the same...

What's new?

AST

All new AST nodes and integration of sub-nodes are complete.
New synthetic node type MixedNode, is created when we have a node whose Symbol has multiple declarations AND multiple base types. So if we merge a class, interface, and function, for example. (Merging with the same base type results in a regular node, ie. ClassNode)
A significant re-design has been done on existing AST, including splitting ObjectNode to -> ClassNode, InterfaceNode, and ObjectNode (represents anonymous object type)
New function nodes: FunctionNode, FunctionParameterNode, CallSignatureNode, MethodSignatureNode, ConstructSignatureNode, FunctionSignatureNode
Bitwise flags have also been updated and can assist in identifying node's contents. (For example, you can identify the base types used in a MixedNode via flags & typeFlags.

json-schema

enum added to tsTypes

Updates/Added for functions/methods:

/**
 * Function signature(s)
 * Valid for:
 *   $tsType: 'function'
 */
$functionSignature?: OneOrMore<FunctionSignature<TDefinition>>

/**
 * Function call signature(s) (specified in type/interface)
 * Valid for:
 *   $tsType: 'interface' | 'type'
 */
$callSignature?: OneOrMore<Exclude<FunctionSignature<TDefinition>, 'name'>>

/**
 * Construct signature(s) (ie. new (): T)
 * Valid for:
 *   $tsType: 'interface' | 'type' | 'class'
 */
$constructSignature?: OneOrMore<Exclude<FunctionSignature<TDefinition>, 'name'>>

/**
 *
 * Valid for:
 *   $tsType: 'interface' | 'class'
 *   type: 'object'
 */
methods?: Record<string, { signature: OneOrMore<RequireSome<FunctionSignature<TDefinition>, 'name'>> }>

Build System

I was still facing some lag issues. I went through a range of different options to mitigate, but overall, I ended up adding a new mode to ts-patch which allows me to 'transform' the actual Program instance that tsc uses.

This allowed me to entirely omit my large options config file which contained all of the heavy mapped types from the Project.

During compilation, the build system plugs into TSC and intelligently determines if a new options source file needs to be generated via watching relevant source files. If it's triggered, it will generate and write the file, then update the Program before TSC emits.

This means:

Generated source is built automatically during compilation
No initial errors from missing generated files
No more lag! 🎉

What's left?

Finish symbol-specific parsing
Add basic test

Because of the extra, extra, extra delays (#compiler-life) rather than make everyone wait for all of Dominik's tests to pass, I'm going to build a smaller isolated test which will suffice to work out any major kinks in the new code and then take it live.

Not to worry - I will still be implementing all of the standard tests and will not switch my work schedule until they're all passing, but hopefully it will help take it public more quickly!

All in all, I'm still stuck deferring important for-profit work until this goes live, so believe me, I'm motivated 😅

Will post another update next weekend, (fingers crossed) hopefully with news that it's ready!

timini commented 4 years ago

Great news, keep up the good work!

nonara commented 4 years ago

Hi folks! Sorry it's been awhile since the last update. I had to spend some time doing maintenance and updates on another library.

That said, there are updates on this front also.

Declaration merging is integrated!

New nodes are created and everything is integrated into the parser. Modifying for symbol parsing brought up some interesting scenarios, but all is complete now.

Extras

readonly modifier and ReadonlyArray support is added

Updates

One thing that's bothered me for awhile has been the AST Nodes. They weren't necessarily true AST nodes, in the sense that some represented TypeScript and some were more toward pure JSON. This and the nodes' flags bitwise enums were not well thought out. (redundant in some cases, etc)

In light of finishing all possible Nodes for TS, I've been able to re-think this a bit.

This sort of mashup between the two was a bit confusing and also limiting, in a sense. The typeFlags was overcrowded and redundant, replicating information already easily able to be discerned by the node Kind or origin.

Ultimately, I've arrived at the realization that there is more value in separation between the AST layer and the TypeScript parser. In other words, the AST should be able to represent type information in a language-agnostic manner whenever possible in terms of Nodes, and the typeFlags should be represent an entirely language-agnostic way of identifying the type it represents.

The end result is the start of a scalable, universal AST for Types that compile to JSON. This opens up the possibility to add more 'parsers' for more languages as we move forward.

Updated Structure (de-coupled AST)

A SchemaNode for every type (some will be universal like FunctionNode, some will be specialized per-language like TypeAliasNode)
typeFlags bitwise enum represents language-agnostic underlying type (ie. primitive, class, interface, etc) and possibly modifiers (ie. paramerized, mixed)

Note: Any non-exposed TS specific type info (like whether the origin for TrueSchemaNode is any or unknown) is still accessible through the origin property, which is the originating ts.Symbol.

Left TODO

The changes aren't major from a code perspective. Mostly a few semantic changes in enums.

1) Update flags enum

Primitives and Literals get PrimitiveKind and LiteralKind enums (frees up much needed space in typeFlags 2) De-couple SchemaNode from direct links to anything TS-parser specific (mainly only the TypeWalkerContext, which has little impact) 3) Initial test to ensure all possible nodes are created and render properly 4) Finish implementing the logic for ParseOptions (deferring most to after releasing src) 5) Implement Rules (deferring to after releasing src)

domoritz commented 4 years ago

Nice. Do these changes also mean that I can use typescript to generate something other than json schema? For example, we generate Altair from the generated Vega-Lite json schema right now but I wonder whether we can go directly from ts to python interfaces.

nonara commented 4 years ago

@domoritz Actually, yes! That would certainly be possible by using the AST Nodes.

That brings up an interesting point. Right now each node has a render method which performs the compilation to JSON schema, but it would probably be even better to entirely de-couple from JSON as well. Then parser and render functions could be hosted in separate modules (packages) on a per-language basis.

That would mean the AST was a true language-agnostic abstraction which represented just type-ish parts of other languages. I love that!

nonara commented 4 years ago

Short update

This will be the last update I post here until it's compiling TypeScript to JSON

The project has finally taken its final form. We also have a name, domain, and orgs on npm and GH.

We have opted to make it entirely language agnostic and have created a new universal AST structure which should support every conceivable type in any language. (more detail in that on the thread linked below)

I imagine that most just want to use the tool to transform TS types to JSON Schema, so I won't bore people with the updates. There are many coming at rapid pace as all of the components get migrated in.

However, if anyone here is interested in the development process:

I've now made the source base live (parser and compiler aren't migrated in yet, but will be very soon).
I've created a thread to share updates as it happens

Invitation

It's been a long six months and has been solo all the way. I'd love it if a few people want to be involved in discussion on the design points for this system. If you or anyone you know would like to be a part of making the final decisions on what I hope to be a widely used platform, I'd really appreciate it. I don't want to force anyone to take an interest, and I'm happy to wrap it up solo, but I'd really like it if anyone wanted to help bounce ideas around. The goal is to to make sure that the final design decisions are well-thought out in order to make sure it scales well and provides a great end-user experience!

If thats you, join the discussion thread, linked below.

If that's not you, that's completely okay! Hold on - I know it's been a long time coming, but the usable tool is just about here.

Links

Repository: https://github.com/crosstype/crosstype
Discussion: https://github.com/crosstype/crosstype/issues/4
Twitter: https://twitter.com/crosstypeorg
Site: https://www.crosstype.org/ (placholder for now)

domoritz commented 4 years ago

@nonara when do you plan to upload the code to support JSON schema generation? I would love to take it for a spin.

nonara commented 4 years ago

@nonara when do you plan to upload the code to support JSON schema generation? I would love to take it for a spin.

New AST + tests are done. I'm migrating the TS type parser in the next work session. That will give you a look at the generated AST for all TS types. After that, I'll port the JSON compiler, then you'll be able to see the full transition!

To allow people to start playing with it sooner, I'll be migrating all of the parse and compile options (tags, etc) after.

domoritz commented 4 years ago

Thanks for the update. It sounds like this won't be ready for a while so I will fix https://github.com/vega/ts-json-schema-generator/issues/446 here instead of moving over yet.

cspotcode commented 3 years ago

I hope this is ok to ask here. We're using schema generation and it doesn't support the unknown type. I assume schemas should treat it identically to any.

Should I send a PR here to add support? Or to the crosstype project?

domoritz commented 3 years ago

The cross type project isn't ready and there is no timeline for json schema support so I would say send a pull request here.

cspotcode commented 3 years ago

Is it safe to assume that the cross type project will never be finished and instead refocus on merging the big 3 schema generation libraries, as proposed here? https://github.com/YousefED/typescript-json-schema/issues/295

I think it may be helpful to restart this discussion and decide how best to merge the big 3 and pay off tech debt.

For the sake of the TS ecosystem, I think it is important to point out a fundamental flaw with the approach taken in the cross-type project:

It was not really open-source. What I mean by that is, although the author often asked for help, the code was closed-source for a very long time. I can understand pros and cons to this approach, but it meant that too many potential contributors were encouraged instead to sit back and be very excited in this thread. (you can see this from all the "I can't wait" comments above)

As is so often the case with software development, I believe it's better to iterate on what we already have, and make breaking changes as necessary to eliminate tech debt. We can still reach the same end-state -- having a well-architected, feature-complete generator -- but we'll be better off along the way, because we'll always have something that works as we continue to refine, improve, refactor, and add features.

If this approach sounds acceptable to others, I can start a new issue with a checklist of tasks based on https://github.com/vega/ts-json-schema-generator/issues/101#issuecomment-497818950

domoritz commented 3 years ago

Building and maintaining open source is difficult and it's often not part of our main jobs. I am grateful for the ideas this thread generated and I hope we can integrate what we learned into the next steps.

I'd be happy for someone to combine the best of the different schema generators as long as it supports the test cases in this repo (or make reasonable adjustments). It would be important to set up a sustainable maintenance model as well.

cspotcode commented 3 years ago

If anyone would like to discuss this realtime, I am usually hanging out on the TypeScript Discord. We can create a dedicated channel or thread for JSON schema generation.

https://discord.gg/typescript

domoritz commented 3 years ago

I joined the discord but I prefer keeping decisions and important discussions on github as it's easier to search.

cspotcode commented 3 years ago

Agreed, and realtime chat tends to have a lot of repetition, because newcomers cannot easily see what's already been discussed.

cspotcode commented 3 years ago

I'm reading through the chatlog, attempting to make a short-list of issues.

There was debate between 2x approaches for getting types: using the typechecker API, or parsing TS ASTs. The typechecker API made it difficult to preserve type aliases.

Does anyone know if this has been addressed by recent changes in TS 4.2 which preserve type aliases in more situations? https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-2.html#smarter-type-alias-preservation

domoritz commented 3 years ago

I don't know but if we can avoid parsing the ast, it will make the code a lot easier and more maintainable. Maybe you can take a stab at a prototype (paying attention to aliases) and see how far you get.

cspotcode commented 3 years ago

Is there a spec for the property names used in the "definitions" object? For example, supposing someone wants to create a schema that includes every type exported from all-schemas.ts, are there some guarantees that the schema for named export Foo will be found at "definitions" "Foo"?

cspotcode commented 3 years ago

Thoughts on using typedoc's type extraction engine to power schema generation? Seems like typedoc has more active maintenance and has better kept pace with typescript's type system, using the typechecker for more of the extraction work than this library. We are fairly active on Discord.

domoritz commented 3 years ago

I don't care what we use under the hood as long as we can generate a good schema. I love the idea of using something well maintained as the basis.

What's the benefit over using the typescript compiler?

cspotcode commented 3 years ago

A few reasons I'm thinking about: motivation, ability to chat with other developers, and avoiding duplicated effort.

I talk to the maintainer and a few other team members pretty regularly on Discord, so it's easier to discuss design decisions, get feedback, we avoid duplicated effort, and it's more motivating to work on typedoc. It's fun to chat and share progress.

Using the compiler API isn't free: it requires understanding the correct way to use it, understanding the quirks, understanding the gotchas to avoid. Typedoc already does that. It uses the typechecker as much as possible except for a few places where it can't. And the typedoc team is available to explain it.

Typedoc handles exports as you would expect, as opposed to this library which exposes internal types, cannot handle multiple types with the same name, and does not handle alias exports. Typedoc needs to accurately document typescript modules, meaning that the extracted type information is more intuitive: it matches the code one-to-one.

Typedoc has some extra features which might be useful in the future. It can extract type info to a JSON dump, then use that export to render docs multiple times. It might improve performance being able to extract types once and then render multiple schemas.

domoritz commented 3 years ago

That all sounds fantastic. Do you want to try making a POC?

cspotcode commented 3 years ago

I want to ask about this specifically, since it may be a breaking change if we go with typedoc:

Typedoc handles exports as you would expect, as opposed to this library which exposes internal types, cannot handle multiple types with the same name, and does not handle alias exports.

This library exposes non-exported types in the schema, using their internal, non-exported names. But I'm not sure why that is necessary, and I'm not sure it is compatible with typedoc's extracted reflections. Do you know if the requirement to expose internals is documented anywhere? Do you know if the identifiers assigned to "definitions" is specced anywhere?

I'd like to propose a simpler way to tell the schema extractor which schemas to extract: The user writes a single file, schemas.ts, which exports one or more types using named exports. The schema generator is given this file as an entrypoint and will emit a single schema containing a "definition" for every type in schemas.ts, with a name matching the named export. All other "definition"s will have verbose, fully-qualified names that include their sourcefile's path to avoid name collisions. If the user wants a top ref, they add a default export.

This pushes configuration into the TypeScript language. schemas.ts can re-export types from elsewhere in a codebase, so it gives you full control over the included schemas. There is an intuitive one-to-one between identifiers in TS and "definition" names in the schema. And name collisions are handled by the TS language.

domoritz commented 3 years ago

I think it can be nice to use internal aliases to name definitions in the schema if we need them (e.g. when we have a recursive data structure). I don't think we usually expose internal aliases otherwise, do we?

I'd like to propose a simpler way to tell the schema extractor which schemas to extract: The user writes a single file, schemas.ts, which exports one or more types using named exports. The schema generator is given this file as an entrypoint and will emit a single schema containing a "definition" for every type in schemas.ts, with a name matching the named export. All other "definition"s will have verbose, fully-qualified names that include their sourcefile's path to avoid name collisions. If the user wants a top ref, they add a default export.

Where would they add a default export? In the source file or in schemas.ts?

I do like that we would avoid the challenge of duplicate types and make it very explicit what types get exported. However, it's not how typedoc works and some users may get confused why they don't see some types. Maybe it's okay if people already have the relevant types exported in their index.ts but I worry that some types might get very deep.

My only use case for this library right now is https://github.com/vega/vega-lite and it would be a lot of work to export all the relevant types again. Do you have a suggestion for how I could reduce the necessary work?

cspotcode commented 3 years ago

I don't think we usually expose internal aliases otherwise, do we?

I'm thinking of this flag:

-e, --expose <all|none|export>
    all: Create shared $ref definitions for all types.
    none: Do not create shared $ref definitions.
    export (default): Create shared $ref definitions only for exported types (not tagged as `@internal`).

It's tough to tell how that flag is meant to behave, but does it fail if 2x non-exported types in different files have the same identifier? I'm fine with creating shared ref definitions, but the tool should ensure they do not have naming conflicts, and the names do not need to follow any particular rules. They can be made human-readable, but there do not need to be any guarantees about their name. If the user wants a type to have a guaranteed "definition" name, they can achieve that by exporting it from schemas.ts

Where would they add a default export? In the source file or in schemas.ts?

In schemas.ts

domoritz commented 3 years ago

Where would they add a default export? In the source file or in schemas.ts?

In schemas.ts

I think you meant a normal export then, not a default expert (as there can be only one). Right?

cspotcode commented 3 years ago

The idea is that the root $ref is determined by the default export, if it exists. Any other named exports will be guaranteed to reside at their name in "definitions". For example:

export {Foo as default};
export {Bar, Baz, Biff};

In the emitted JSON schema:

The root "$ref" is guaranteed to refer to Foo. You can use the emitted JSON schema as-is to validate that an object matches Foo.

"definitions": {"Bar" is guaranteed to be the definition for Bar. There will likely be many other definitions in the emitted JSON, but their names are generated by the tool and may be more verbose, may include filenames encoded in some form, and naming collisions will be avoided.

This means that you can trivially modify the root $ref to use a single schema to validate against Bar, Baz, or Biff because their definitions are guaranteed to reside at those names in "definitions"

domoritz commented 3 years ago

There will likely be many other definitions in the emitted JSON, but their names are generated by the tool and may be more verbose, may include filenames encoded in some form, and naming collisions will be avoided.

Oh, I missed the part about other definitions being included potentially. I see. So definitions that are exported will have a predictable name and any other aliases or types could have arbitrary names (e.g. to include the path). That's a good idea. I think it makes sense overall.

maneetgoyal commented 3 years ago

Hi all, I have been using Zod for input validation in some of my projects. Since, the discussion here seems to be related to producing JSON schema from TS definitions, thought of sharing ts-to-zod and zod-to-json-schema libraries. Combining both, I think we can do TS --> Zod --> JSON Schema. Would love to know what the developers on this forum think about the limitations of these tools.

domoritz commented 3 years ago

That's interesting. I wonder how flexible Zod is, though. Does any information get lost in the intermediate representation?

cspotcode commented 3 years ago

My immediately thought: does it support customization of the schema with JSDoc @tags? Currently we can customize the JSON schema:

/** @minimum 10 */ foo: number; to set a JSON Schema "minimum"
/** @TJS-default {10} */ foo: number; to set JSON schema default
/** @TJS-type {integer} */ foo: number; to override the schema type to integer instead of number

maneetgoyal commented 3 years ago

Does any information get lost in the intermediate representation?

Running some tests currently. So far, it seems like it can't handle circular dependencies in type definitions. Getting the following warning while running npx ts-to-zod some-src.ts some-dest.ts:

›   Warning: Some schemas can't be generated due to circular dependencies:

maneetgoyal commented 3 years ago

does it support customization of the schema with JSDoc tags?

As per their docs, they support only 6 of the JSDOC keywords.

From their docs:

// source.ts
export interface HeroContact {
  /**
   * The email of the hero.
   *
   * @format email
   */
  email: string;

  /**
   * The name of the hero.
   *
   * @minLength 2
   * @maxLength 50
   */
  name: string;

  /**
   * The phone number of the hero.
   *
   * @pattern ^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$
   */
  phoneNumber: string;

  /**
   * Does the hero has super power?
   *
   * @default true
   */
  hasSuperPower?: boolean;

  /**
   * The age of the hero
   *
   * @minimum 0
   * @maximum 500
   */
  age: number;
}

// output.ts
export const heroContactSchema = z.object({
  /**
   * The email of the hero.
   *
   * @format email
   */
  email: z.string().email(),

  /**
   * The name of the hero.
   *
   * @minLength 2
   * @maxLength 50
   */
  name: z.string().min(2).max(50),

  /**
   * The phone number of the hero.
   *
   * @pattern ^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$
   */
  phoneNumber: z.string().regex(/^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$/),

  /**
   * Does the hero has super power?
   *
   * @default true
   */
  hasSuperPower: z.boolean().default(true),

  /**
   * The age of the hero
   *
   * @minimum 0
   * @maximum 500
   */
  age: z.number().min(0).max(500),
});

M-jerez commented 2 years ago

Hi first of all thanks for the good work to all the authors of these libraries.

Secondly, If there is gonna be a new version with breaking changes, the jsdoc annotations could/should be ditch in favour of decorators.

domoritz commented 2 years ago

Can you explain why?

M-jerez commented 2 years ago

@domoritz Decorators are the official way to add metadata to typescript and es6. jsDoc annotations are a workaround used before decorators existed, jsDoc annotations should be used only/mostly for documentation.

Also most of the libraries related to data mapping , orm etc, are using decorators now. Working with decorators might also be easier more flexible and extensible than jsDoc annotations.

vega / ts-json-schema-generator

Future of schema generators #101

Meanwhile

Update

What's left?

What can I see?

Source Outline

How to look at it

What's next?

Update

What's new?

AST

json-schema

Build System

What's left?

Declaration merging is integrated!

Extras

Updates

Updated Structure (de-coupled AST)

Left TODO

Short update

Invitation

Links