search

Azure / azure-rest-api-specs

The source for REST API specifications for Microsoft Azure.
MIT License
2.68k stars 5.11k forks source link

[TypeSpecValidation] Improve performance of `-CheckAll` #28490

Open mikeharder opened 7 months ago

mikeharder commented 7 months ago

Running TypeSpecValidation.ps1 -CheckAll is getting slow due to the number of specs. Potential improvements:

  1. Validate specs in parallel
    • Validating specs within same spec family can be tricky due to shared code and configurable swagger output folders, but it might be safe if we parse the output from tspconfig.yaml and/or copy spec families to temp folders.
  2. If still need more parallelism, shard across agents
    • Specs named A to M
    • Specs named N to Z
  3. Improve efficiency of TSV checks
    • Call tsp compile and tsp format in-process from Node, rather than shelling out every time
danieljurek commented 1 month ago

To start:

Parallelize by "top level" spec folder (everything in there running sequentially is "safe"). This can be started in PowerShell and moved to Node.

mikeharder commented 1 month ago

Next step is to remove the dependency on git from tsv as much as possible, since some (most? all?) git commands are not safe to execute in parallel in the same repo.

Compile and Format can:

  1. Copy target spec to /tmp/tsv-scratch-[guid]/spec
  2. Symlink node_modules from current npm prefix to /tmp/tsv-scratch-[guid]/node_modules.
  3. Run tsp and prettier commands against scratch folder
  4. Compare scratch to target using git diff --no-index spec /tmp/tsv-scratch-[guid]/spec, which should hopefully be threadsafe and just a convenient way to diff two folders, regardless of whether either is in a git repository
    • If we wanted to completely remove the dependency on git, we could use a package like https://www.npmjs.com/package/dir-compare. But git diff presents exactly the information we want to show users with no extra work.
  5. By default, copy updates from scratch to target, to preserve back-compat with existing in-place update behavior
  6. Add flag like --validate-only to not copy from temp back to repo, which can be used in CI when running TypeSpecValidationAll, to make the tool more performant and completely threadsafe.