PDE-4968 feat(schema-to-ts) - Introduce the Schema-to-TS compiler

[tkcranny]

[!WARNING]
This is rather extensive and being contributed without much knowledge of the DevPlatform's architecture or process. Any feedback or suggestions, no matter how sweeping will be kindly appreciated.

This introduces the new schema-to-ts project in the packages/ directory that compiles typescript definitions from packages/schemas and outputs packages/core/types/zapier.generated.d.ts. This solution has been explored in the past but has abandoned until now, owing to the difficulties expressing function and promise syntax between JSON Schema and TypeScript. See #8 and #260 for further details.

This is a cleaned up and expanded version of a Hackweek Project from Team Integrations in Q2 2024 (coda).

How it works

• It is a TypeScript module itself, built with yarn build.
• It's run via yarn generate-types, which will run dist/cli.js (once built). This CLI has a number of flags and options if needed.
• It reads ../schema/exported-schema.json for its input (by default).
• It outputs ../core/types/zapier.generated.d.ts (by default).
• The generated code is referenced in a unified export with the existing types (previously from index.d.ts, now zapier.custom.d.ts), and are all intended to be bundled in the ZPC distribution on npm for downstream developers.
• It has precommit and git-add scripts in its package.json to build the generated types on commits via husky & lerna, similar to how cli and schema build their docs and output artefacts.
• Internally, we use the internals from the json-schema-to-typescript (npm) library. We choose to render our own TypeScript output with this intermediate result due to some unpleasant limitations and to correctly patch in real function definitions and links to public documentation.
  • See this issue opened by a former Zapien's ticket on the same repo for the same reason.

Potential Issues

• As this is an in situ tool to generate the types per-commit, I don't foresee it needing to built and pushed to NPM like the other four packages. Therefore its place in packages/ or even this repo itself may not be the correct location.
• It is an esm package and uses TypeScript and Vitest, so it's different yet again from the existing CJS packages.
• I know one dependency states it requires node 18 (marked, for reflowing the schema markdown docs), and some other parts of the DevPlatform state a max of v16, so node versions may yet prove problematic.
• It's non-standard and non-trivial to go from our JSON Schemas to TypeScript. While it has enormous benefit to the engineers, contractors, and partner developers that make integrations, I'm cognizant of it being disruptive and the potential future maintenance burden to drop in like this.

See Also

• The Team Integrations Hackweek Coda on Typing
• Closes #8 from 2016
• The (not-merged, closed) PR that attempted it first: #260
  • Comment on why #260 was closed and #8 was reopened in 2021: https://github.com/zapier/zapier-platform/issues/8#issuecomment-827972091
• David Brownman's issue thread from 2020 on the limitations of json-schema-to-typescript: bcherny/json-schema-to-typescript/issues/334

[tkcranny]

I've made some considerable changes today. Notably they include:

• End-to-end tests of the generated types with a pretty standard integration structured app.
  • The main test file is in core/types/index.test-d.ts.
  • This is then tested by the tsd typescript definition checker (GitHub).
  • It is invoked via yarn type-tests, and subsequently via yarn test in the core package.
• I've added BeforeRequestMiddleware and AfterResponseMiddleware types to zapier.custom.d.ts module.
  • These are then referenced and replace the plain MiddlewaresSchema for the respective types via a transformer for each.
• I've gotten a lot more tests working. Now it's just a matter of getting the typescript template to pass it's smoke test.
  • I'm having a lot of trouble getting more details out of the zapier init function this is running. I think that output is itself for some spawned subprocesses it's running.

[tkcranny]

@rnegron:

In terms of the developer experience of generating the typing declarations file, I saw the precommit script in package.json but it doesn't seem to yarn (install the package requirements) beforehand. I suppose for most contributions to this repo, changes would not need to be made to this package and therefore it should be almost transparent that the typing is auto-generated before a commit happens in zapier-platform-core (and zapier-platform-schema!) What about adding a lifecycle step to the core (and schema?) package to ensure that schema-to-ts is ready to run on changes to that package? There are a lot of available lifecycle options for npm scripts 😵.

I love this idea. I've tried a couple of things to this effect. What I've found by putting in lifecycle hooks into the Core/Schema packages to ensure schema-to-ts is installed though breaks both the node 16 tests, as well as the smoke-tests when the packages are built and used on their own with schema-to-ts missing from the expected location.

Fortunately yarn install from the root of the repo will prepare schema-to-ts's own node_modules in preparation for the transparent commit hooks that generate the output declarations.

Therefore the biggest issue I see is that people with this repo already checked out (i.e. you folks) will just need to run yarn from the root to get this tool to work. Fresh installs ought to be fine because of this too.

Is that acceptable for you?

[xavdid]

@tkcranny Congrats on getting this out the door! I was (apparently) still subscribed to the original issue so I got the email that it had been closed. This is an awesome approach; color me quite impressed! 🎉