Easy generation of multiple types inside same folder

[sp88011]

Feature description

Imagine types are stored in files following this folder structure:

src > parsers > a.ts
src > parsers > b.ts
....
src > parsers > x.ts

Input

yarn ts-to-zod src/parsers/* src/nowIcanValidateEverything.ts

Output

nowIcanValidateEverything.ts has all generated zod types found in any of the files with the path src/parsers/

[fabien0102]

It's a bit harder than it looks since you can have name conflicts in the files (like two export type A = … in a.ts and b.ts) You should be able to do this easily with a bit of glue, ts-to-zod has a programmatic API, you can see an example here: https://github.com/fabien0102/ts-to-zod/blob/8d3b297e4131ef105bfe155b038e4b4f76a7d5fc/src/cli.ts#L249-L256

(you can import generate from ts-to-zod)

[brauliodiez]

Mmm... thinking loud, in my case we use the following naming convention:

Model (entities) files, follow the convention: name.model.ts

Zod schema files, follow the convention: name.zod.ts

So in theory I could just implement some script that traverses all subfolders under src and invoke ts-to-zod does that makes sense?

Thanks for you great tool

[greenpixels]

I actually did the same thing that @brauliodiez suggests, so I thought I'd add a comment here. I use zod and ts-to-zod for a little web-game side-project using socket.io communication.

I wrote a script for iterating through sub-directories, searching for typescript files and then creating a zod-schema for it using ts-to-zod:

import { glob } from 'glob'
import { exec } from 'child_process'
import * as path from 'path'

const GLOB_PATTERN = 'dtos/*.ts' // Where to search for typescript definitions?
const GLOB_IGNORE_PATTERNS = ['node_modules/**', '**/*.zod.ts'] // What should be ignored? (*.zod.ts to prevent recursion)
const VERBOSE = false // Should native ts-to-zod logs be printed?
const FILES = await glob(GLOB_PATTERN, { ignore: GLOB_IGNORE_PATTERNS })

FILES.forEach((filePath) => {
    const { base, dir, name } = path.parse(filePath)
    const FILE_PATH_NORMAL = path.join(dir, base)
    const FILE_NAME = name
    console.log(`Generating Zod-Files for ${FILE_NAME}`)
    exec(
        `npx ts-to-zod ${FILE_PATH_NORMAL} ${FILE_PATH_NORMAL.replace('.ts', '.zod.ts')} --config=${FILE_NAME}`,
        (error, stdout, stderr) => {
            if (error && VERBOSE) {
                console.error(error.message)
            }
            if (stdout && VERBOSE) {
                console.log(stdout)
            }
            if (stderr && VERBOSE) {
                console.error(stderr)
            }
            console.log(` ✔ Generated Zod-File for ${FILE_NAME}`)
        }
    )
})

This obviously also matches normal .ts files that are not Types or Interfaces, so if your folder structure has types and modules mixed, you'd need to adjust the pattern, structure or the file names. :)

And you also need to map all your affected files in the correct import-dependency order inside of the ts-to-zod.config.js by hand in order to prevent issues.

/**
 * ts-to-zod configuration.
 *
 * @type {import("ts-to-zod").TsToZodConfig}
 */
module.exports = [
    { name: 'Vector2DTO', input: './dtos/Vector2DTO.ts', output: './dtos/Vector2DTO.zod.ts' },
    { name: 'EntityDTO', input: './dtos/EntityDTO.ts', output: './dtos/EntityDTO.zod.ts' },
    { name: 'ProjectileDTO', input: './dtos/ProjectileDTO.ts', output: './dtos/ProjectileDTO.zod.ts' },
    { name: 'PlayerDTO', input: './dtos/PlayerDTO.ts', output: './dtos/PlayerDTO.zod.ts' },
]

I'm pretty sure you could even, under the right circumstances and project structure, create the configuration on the fly by creating a preprocess step in which you try to figure out the order of the files based on whether or not they have imported another scanned file or something like that. Probably way easier said than done tho. :)

Anyway, really cool project!

[tvillaren]

Thanks for sharing @greenpixels !

Yes, I believe the complexity is in ordering the files based on their imports (and I was actually thinking about generating a config file on-the-fly based on the right ordering). Once we have it, the rest is already implemented.

[pkyeck]

@tvillaren any progress/updates on the ordering?

[tvillaren]

@pkyeck Unfortunately not on my side, haven't had the personal time to put into it this month.

[pkyeck]

ok, came up with a solution for my very special case where I know where to look for types/imports:

Quick & dirty updated version of the aforementioned script

import { execSync } from 'node:child_process'
import * as fs from 'node:fs'
import * as path from 'node:path'

import { glob } from 'glob'

const topologicalSort = (dependencies: Record<string, Array<string>>) => {
  const visited = new Set()
  const result: Array<string> = []

  // Function to perform DFS
  const dfs = (item) => {
    if (visited.has(item)) {
      return
    }
    visited.add(item)

    for (const dep of dependencies[item]) {
      dfs(dep)
    }

    result.push(item)
  }

  // Perform DFS for each item in the graph
  for (const item in dependencies) {
    if (!visited.has(item)) {
      dfs(item)
    }
  }

  // get the correct topological order
  return result
}

const main = async () => {
  const GLOB_PATTERN = '**/*.types.ts' // Where to search for typescript definitions?
  const GLOB_IGNORE_PATTERNS = ['node_modules/**', '**/*.zod.ts'] // What should be ignored? (*.zod.ts to prevent recursion)
  // const VERBOSE = true // Should native ts-to-zod logs be printed?
  const FILES = await glob(GLOB_PATTERN, { ignore: GLOB_IGNORE_PATTERNS })

  const dependencies: Record<string, Array<string>> = {}

  const configEntries: Array<{ name: string; input: string; output: string }> = []

  FILES.forEach((filePath) => {
    const { base, dir, name } = path.parse(filePath)
    const FILE_PATH_NORMAL = path.join(dir, base)
    const FOLDER_PATH = FILE_PATH_NORMAL.split('/').slice(0, -1).join('/')
    const FILE_NAME = name
    console.log(`Generating Zod-Files for`, { FILE_PATH_NORMAL, FILE_NAME, name })

    const data = fs.readFileSync(FILE_PATH_NORMAL)
    const content = data.toString()
    const regex = /import .+ from '(.+\.types)'/g
    const imports = [...content.matchAll(regex)]
    dependencies[FILE_PATH_NORMAL] = imports.map(([, importPath]) => `${path.join(FOLDER_PATH, importPath)}.ts`)
  })

  const orderedFiles = topologicalSort(dependencies)
  orderedFiles.forEach((f) => {
    configEntries.push({ name: f, input: f, output: f.replace('.types.ts', '.zod.ts') })
  })

  const contentOfConfig = `module.exports = [
  ${configEntries.map((i) => JSON.stringify(i)).join(',\n  ')}
]`
  fs.writeFileSync('ts-to-zod.config.js', contentOfConfig)

  // build all configs at once
  // const cmd = `npx ts-to-zod --all`
  // console.log(`Running: ${cmd}`)
  // try {
  //   const buf = execSync(cmd)
  //   console.log(` ✔ Generated Zod-Files`, buf.toString())
  // } catch (error) {
  //   console.error(error)
  // }

  // build zod files one by one
  orderedFiles.forEach((f) => {
    const cmd = `npx ts-to-zod --config=${f}`
    console.log(`Running: ${cmd}`)
    try {
      const buf = execSync(cmd)
      console.log(` ✔ Generated Zod-File for ${f}`, buf.toString())
    } catch (error) {
      console.error(error)
    }
  })

  console.log(`Found ${FILES.length} files`)
}

main()
  .then(() => {
    process.exit(0)
  })
  .catch((error) => {
    console.error(error)
    process.exit(1)
  })

Don't really know if I should run these one by one or just with the --all parameter once we created the config 🤷‍♂️ Does this make a difference?

[tvillaren]

Thanks for sharing @pkyeck 🙏

The --all parameter runs all generation within a Promise.all method with each generation having a few file-reading async calls + spawning a Worker for validation purpose. So I guess that with this param, it could happen (through race condition) that the generation of A referenced in B finishes after the generation of B, which would lead to validation failing.

[pkyeck]

The --all parameter runs all generation within a Promise.all

Good to know. Then I should stick to the running them one by one with --config=<name> 👌

[t-animal]

This should be a good starting point for sorting files (we're using it in our project):

const dependencyTree = require("dependency-tree");
const glob = require("glob");
const path = require("node:path");

function sortFilesByGenerationOrder(filesToSort, srcFolder, tsConfig) {
    // JS sets are ordered, so we can rely on it for iteration and deduplication
    const dependencies = new Set();

    const visited = {};
    for (const file of filesToSort) {
        const newEntries = dependencyTree.toList({
            filename: file,
            directory: srcFolder,
            tsConfig,
            visited,
        });
        newEntries.forEach((it) => dependencies.add(it));
    }

    return [...dependencies];
}

const srcFolder = path.join(__dirname, "src");
const files = glob.globSync(path.join(srcFolder, "*.ts"), { absolute: true });
const ordered = sortFilesByGenerationOrder(files, srcFolder, "./tsconfig.json");

/**
 * ts-to-zod configuration.
 *
 * @type {import("ts-to-zod").TsToZodConfig}
 */
const config = ordered
    .map((f) => f.replace(srcFolder + path.sep, "").replace(/.ts$/, ""))
    .map((file) => ({
        name: file,
        input: `src/${file}.ts`,
        output: `src/schemas/${file}.zod.ts`,
    }));

module.exports = config;

/**
 * This is a hack until https://github.com/fabien0102/ts-to-zod/issues/151 is implemented
 * We're running this file to generate the config entries in order. When the issue has been
 * closed, all code after this comment can be removed and the built-in mechanism can be used.
 */

const { argv } = require("node:process");
const { spawnSync } = require("node:child_process");
const isRunningAsScript = __filename === argv[1];

if (isRunningAsScript) {
    for (const configEntry of config) {
        console.log(`Generating ${configEntry.name}`);
        spawnSync("npx", ["ts-to-zod", "-c", configEntry.name], { stdio: "inherit" });
    }
}

Generate using node ts-to-zod.config.cjs (or node ts-to-zod.config.js when using a commonjs package).

PS: When using dependencyTree instead of dependencyTree.toList, one could even find unrelated groups of files and generate the groups in parallel using Promise.all, which would speed up generation. I would say that's a future optimiziation, though.