microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
100.68k stars 12.44k forks source link

Proposal for merging array or object options in tsconfig file #57486

Open sheetalkamat opened 7 months ago

sheetalkamat commented 7 months ago

🔍 Search Terms

"extends", "merge", "compilerOptions", "paths", "outdir", "include", "exclude", "typeRoots", "tsconfig", "ability to merge tsconfig"

✅ Viability Checklist

⭐ Suggestion

Subset of #56436

Today when you extend a config the options written in the final config override the options from the base config. Here is sample of what happens today:

// @fileName: /temp/test/base.tsconfig.json
{
    "compilerOptions": {
        "paths": {
            "@libs/*": ["libs/*"],
            "@project1/*": ["project1/*"]
        },
        "typeRoots": ["node_modules/@types"],
    }
}
// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings"]
    }
}

Here is what it would mean that project1 tsconfig was written as:

// @fileName: /temp/test/project1/computed.tsconfig.json
{
    "compilerOptions": {
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["./typings"],
    }
}

Over years users have asked us for ability to create a final config that adds to base config instead of overriding. This is extremely useful for paths from compiler options.

This would allow to users to write a base config that contains common options that shouldn't have to be copied in all projects and yet allowing projects to specify there additional specific options.

For example in above sample, one would expect a way to write config such that final paths property is:

// @fileName: /temp/test/project1/computed.tsconfig.json
{
    "compilerOptions": {
        "paths": {
            "@libs/*": ["../libs/*"],
            "@project1/*": ["../project1/*"],
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["../node_modules/@types", "./typings"],
    }
}

Here are different proposals that are being considered:

Per Property Specific options

1. merge fields

Add merge as root level config property that specifies options that need to merged instead of completely overwritten in the final config. From the above example writing the project1 config as:

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings"]
    },
    "merge": ["paths", "typeRoots"]
}

2. Inline merge options:

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "merge": {
            // All the options that need to be merged
            "paths": {
               "@environment/*": ["env/*"]
            },
            "typeRoots": ["typings"]
        },
    },
}

3. merge:optionName in instead of just optionsName

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "merge:paths": {
            "@environment/*": ["env/*"]
        },
        "merge:typeRoots": ["typings"]
    },
}

4. Root level merge:OptionsBagName to merge.

From the above example writing the project1 config as:

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "merge:compilerOptions": {
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings"]
    },
}

5. merge-optionsName: true in the options

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "merge:paths": true,
        "merge:typeRoots": true,
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings"]
    },
}

6. Simple placeholder like ${base}: true or <merge>: true

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "paths": {
            "${base}": true,
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["${base}", "typings"]
    },
}

7. Detailed template substitution

This should allow you to control the order esp when extending multiple configs From the above example writing the project1 config as:

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "compilerOptions": {
        "paths": {
            "${0}": true,
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings", "${0}"]
    },
}

Global merge options

8. Add root level merge: true that merges all arrays and objects options:

From the above example writing the project1 config as:

// @fileName: /temp/test/project1/tsconfig.json
{
    "extends": "../base.tsconfig.json",
    "merge": true,
    "compilerOptions": {
        "paths": {
            "@environment/*": ["env/*"]
        },
        "typeRoots": ["typings"]
    },
}

Here is a chart of all options that are list or object type:

Option Category Option Type Order Dependency Eligible for Merge
CompilerOptions paths object mayBe not #44589, #20110 , For merging paths, we set pathsBasePath for resolving relative path, what happens to that when paths are merged
CompilerOptions lib list no #20110
CompilerOptions rootDirs list yes?
CompilerOptions typeRoots list yes Now that we resolve types only from typeRoots, this seems to be important?
CompilerOptions types list no? global types to include
CompilerOptions moduleSuffixes list yes
CompilerOptions customConditions list no
CompilerOptions LS plugins list mayBe
WatchOptions excludeFiles list no
WatchOptions excludeDirectories list no
TypeAcquisition include list no
TypeAcquisition exclude list no
Root files list mayBe #20110
Root include list mayBe #20110
Root exclude list mayBe #20110
Root references object no Transitive, vs listing only direct parent? #30608, #27098

📃 Motivating Example

A way to specify say project specific "paths" mapping without having to rewrite all the mappings from the base config.

💻 Use Cases

  1. What do you want to use this for?
  2. What shortcomings exist with current approaches?
  3. What workarounds are you using in the meantime?
ijxy commented 7 months ago

Can you provide an example of a project structure that this would actually be useful for?

I am having a difficult time thinking of any that isn't extremely convoluted by design (and I don't think those are a problem for TypeScript to solve).

The beauty of tsconfig is that it is pretty much WYSIWYG. Every key will either have the value I specify or the value in the config I extend from with no ambiguity.

RyanCavanaugh commented 7 months ago

OP for this one is #44589 where there are many people complaining that it's not possible. Unfortunately, people are just noting that the issue has been open for X amount of time instead of providing more context on their setups.

SainteCroquette commented 7 months ago

I think main motivation would be mono-repositories, where you would want to share typescript configuration across your repository's projects, which most of the time include paths.

Some real world example with Nx, a popular monorepo management solution :

nx integrated monorepos typically have a root level tsconfig.base.json which is then extended by every app / library / package in the repository from which theses modules will find paths definitions among others: (generated with nx's cli)

./tsconfig.base.json

// define libraries path so that they can be used in other local packages. 
{
  // ...
  "compilerOptions": {
    // ...
    "baseUrl": ".",
    "paths": {
      "@myorg/some-lib": ["libs/some-lib/src/index.ts"],
      "@myorg/some-other-lib": ["libs/some-other-lib/src/index.ts"],
    }
  },
}

./apps/some-app/tsconfig.json

// project specific options 
{
  "extends'": "../../tsconfig.base.json",
  // ...
  "compilerOptions": {
  // theses are local path aliases, only known to this module. needs full path as baseUrl is set in root tsconfig.base.json
    "paths": {
      "@somelocalalias/*": ["apps/some-app/src/somelocalfolder/*"],
    }
  },
}

Current behavior makes maintaining local imports a pain :/ Workarounds do exists but most of the time they involve custom scripts or adding a new dependency which is far from ideal for such uses cases as there is no opinionated way of achieving this.

ijxy commented 7 months ago

@SainteCroquette Still not really understanding. This is the exact problem that a monorepo solves--using packages.

In other words, why would you use inherited tsconfig paths instead of dependencies?

// my-app/package.json

{
  "dependencies": {
    "@myorg/some-lib": "*",
    "@myorg/some-other-lib": "*"
  }
}
// my-app/some/file.ts

import { something } from "@myorg/some-lib";
import { somethingElse } from "@myorg/some-other-lib";

I've used TypeScript monorepos for a relatively long time and never once encountered a reason to use paths instead of dependencies (even tsconfig files can, and I think should, be shared via dependencies).

SainteCroquette commented 7 months ago

I can't speak for other tools but if you follow Nx (which doesnt rely on package name to resolve paths) integrated monorepository guidelines (which is really easy thanks to their cli & plugins) you end up with:

Inside a package, build tools such as webpack or vite can then use a plugin provided by nx to feed them the path options from the local tsconfig. To be able to map correctly to other packages from your mono repository the local tsconfig needs to extends base tsconfig which holds the mapping configuration of the whole repository. Defining a path option for the local tsconfig would then break the whole toolchain. Would this means that nx's choice of relying on tsconfig as 'single source of truth' is against intended usage of tsconfig's path options?

Could you expand a bit on sharing tsconfig files via dependencies and how this would allows you to merge to path configurations together (or is this another way to achieve the same thing ?) ?

antongolub commented 7 months ago

If the config structure is standardized, then the merging rules are too. Option 8 proposes the behavior that is predictable and easy to implement, imo. Let's not complicate the config logic. Templates are a kind of Pandora's box, you may believe me.

ijxy commented 7 months ago

Could you expand a bit on sharing tsconfig files via dependencies and how this would allows you to merge to path configurations together (or is this another way to achieve the same thing ?) ?

@SainteCroquette Package-based monorepos (ie npm workspaces) also work with Nx. The Nx integrated plugin "magic" is completely opt-in. In other words, the need for merging tsconfig paths in such a setup is a direct consequence of the way Nx internals work. As such, it seems more reasonable to me that Nx (in this case) should provide a solution (eg via another plugin to generate tsconfig files with the appropriate paths).

The question is, where does TypeScript draw the line about what it supports out of the box? To me, it feels right that it works with the standard npm/node resolution. Anything beyond that, I think, is likely to lead to endless scope-creep.

Bessonov commented 7 months ago

Just to mention that Webpack uses special '...' syntax to include defaults.

saltman424 commented 7 months ago

@jxdp one issue with TypeScript choosing to rely on npm workspaces/dependencies as the primary supported setup for mono-repos is that it unnecessarily couples TypeScript to node/npm for mono-repos. There are other runtimes for TypeScript, such as Deno and Bun. Personally, I prefer node, but it seems to me that a self-contained TypeScript solution for mono-repos has some advantages.

Now you could take your point more generically, that mono-repos are outside the scope of TypeScript. I.e. the burden of making TypeScript useful in mono-repos is entirely on npm, Nx, or whoever else wants to take it on. One problem with that is TypeScript is already part of the way there. Just using extends and paths allows you to easily create buildable code in a mono-repo. So at best, there is going to be a split burden because TypeScript mono-repos, whether or not they are using a third-party tool like npm or Nx, are sometimes going to use or these properties but then are also limited by them, and thus we will likely continue to see interest in issues like #44589.

As for the question on use cases, I think the Nx integrated mono-repo approach that @SainteCroquette mentioned is a reasonably common one. And I disagree with

the need for merging tsconfig paths in such a setup is a direct consequence of the way Nx internals work

At least in my use case, I use Nx just to orchestrate tasks. My code is able to build and run entirely without Nx. We could easily swap Nx out for a different task orchestration framework and still build and run all the same TypeScript code with all the same tsconfigs. So it would be nice to not have to copy-and-paste paths between those tsconfigs.

In fact, Nx effectively does not intrude on the actual mechanics of sharing code in your project by allowing you to either use npm workspaces (package-based) or TypeScript paths (integrated). In other words, they seem to be treating it as if npm and TypeScript provide two different, valid ways for sharing code in a mono-repo, and they just try to support both.

Also, to this objection:

The beauty of tsconfig is that it is pretty much WYSIWYG. Every key will either have the value I specify or the value in the config I extend from with no ambiguity.

I would argue that the extends property makes the tsconfig precisely not WYSIWYG. Especially, since you can have an indefinite chain of extends. It is not uncommon for me to have to go through a series of extended tsconfigs to find the property I am looking for, only to find it is not specified anywhere, at which point I have to look up the default value. Also, I don't think most of the proposed changes here would meaningfully alter the degree to which tsconfig is WYSIWYG. You would still see what is specified in your tsconfig and then have to look at the extended tsconfig for anything where your tsconfig indicates that it is using the extended tsconfig (i.e. by omitting the property or by explicitly merging it). So as long as a relatively intuitive approach is used for merging, not much changes. And I think the point of this issue is to evaluate how intuitive and useful each of the proposed approaches are.

TLDR

  1. With the extends and paths properties of tsconfigs, it is not hard to create buildable code in a TypeScript mono-repo without any other tooling
  2. This merging feature would make that experience even better, especially for paths, and regardless of what other tooling is used
  3. tsconfigs are already not WYSIWYG
saltman424 commented 7 months ago

Also, to add to the discussion around use cases and preferred merge approach:

Use Case

I would primarily want this for paths, include, and exclude.

I have a TypeScript mono-repo that uses TypeScript for everything (Angular frontend, Node backend, AWS CDK infrastructure, and various shared TypeScript libraries).

I have a series of nested folders for the various parts of the mono-repo, with various layers of tsconfigs (and various tsconfigs for different use cases, like tests, building, linting, etc.). This means there is lots of extending going on (e.g. root/packages/backend/myapp/tsconfig.test.json extends root/packages/backend/tsconfig.test.json extends root/tsconfig.test.json extends root/tsconfig.base.json). I use paths to reference other projects in the mono-repo. The main reason I am interested in this feature is I have some libraries available to everything (in paths from root/tsconfig.base.json) and I have some libraries available only on say the backend (in paths from root/packages/backend/tsconfig.json). Currently, I have to either (1) copy-and-paste the paths from root/tsconfig.base.json into root/packages/backend/tsconfig.json, or (2) move the backend paths to root/tsconfig.base.json and just not import them in the other parts of the mono-repo. I generally opt for (2) because I am worried about the copied paths getting out of sync over time. However, in some cases where I have a ton of paths for a specific part of the mono-repo, I copy the base paths because I don't like cluttering root/tsconfig.base.json.

Fortunately, I am able to use wildcard paths to keep the number of paths pretty small, which makes copying them not as bad. But still not ideal.

I use Nx to orchestrate tasks, esbuild to build code, and tsx to execute scripts. As mentioned above, I don't find Nx to be an important part of this issue, at least for me, as regardless of how I trigger esbuild or tsx, they still work the same way, using my tsconfigs and paths properties to resolve dependencies. Although, I know Nx has various features like dependency detection that could potentially be affected by this feature.

Preferred Approach

In reading about verbatimModuleSyntax, it seems there can be some advantages to just having the developer be explicit rather than trying to guess the desired outcome. So with that in mind, I like template variables.

To answer this question from #57567:

How would you specify the paths of a or b?

  • These are not strings, paths is an object with entries.

Personally, I wouldn't mind changing paths to a list just in this use case. E.g.,:

"paths": [
  "${a}",
  "${b}",
  {
    "@mystuff/*": [ "./*" ]
  }
]

Although, I recognize this could introduce breaking changes for third-party tools that read tsconfigs. Note: paths as an object would still be valid and just wouldn't have merging.

If the list is unacceptable, then you could have a special symbol for merges (e.g. &):

"paths": {
  "&": [ "${a}", "${b}" ],
  "@mystuff/*": [ "./*" ]
}

Or, this might be complicated, but you could use extended paths when they match a given path. E.g.:

"paths": {
  "*": [ "${a}" ],
  "@b/supported/*": ["${b}"],
  "@mystuff/*": [ "./*" ]
}

In this case, all the paths from a are usable, but only the paths from b that match @b/supported/* are usable.

Simple vs Detailed Template Variables

As for the question of simple vs. detailed template variables. I think detailed template variables should very rarely be needed. Most of the time, you would want to use the same property as the one you are modifying (i.e. in paths I want to reference base.paths, in include I want to reference base.include). In theory, there might be use cases where you want to do something like "exclude": [ "${base.include}" ], but that seems strange, and I definitely don't think it needs to be supported.

Also, it would be nice to have a special template variable, maybe ${*}, which represents all extended configs in the order specified in extends. E.g.,:

"paths": [
  "${*}",
  {
    "@mystuff/*": [ "./*" ]
  }
]
SainteCroquette commented 7 months ago

@saltman424 while templates are powerful & flexible and they do solve current problem, they often lacks readability. They might be too much vodoo for that kind of use case.

Personally, I favor comprehensiveness over conciseness. Moreover that would probably increase overall complexity of tsconfig files, going against its straightforward declarative syntax.

Proposals 1, 2 and 3 are explicit in that sense. Not sure how you would handle the case where there is multiple files are extended tho #57567

saltman424 commented 7 months ago

@SainteCroquette that's fair. I mostly just don't want TypeScript to have to continually release refinements and new properties to address every edge case with merging. The main appeal to me with template variables is it gives me a lot of control so if there are edge cases that come up, I can resolve them myself. And the results are just more predictable. I find behind-the-scenes magic great when it works and really frustrating when it doesn't.

With that being said, the default merge logic would probably work for most of my current use cases, so I am not opposed to it. For multiple extends, I would assume the merge logic would just be applied in order of extends. I.e.

{
    "extends": [ "./a", "./b", "./c" ],
    "compilerOptions": {
        "paths": {
            "@mystuff/*": ["./*"]
        },
    },
    "merge": [ "paths" ]
}

would use paths = merge(merge(merge(a, b), c) { "@mystuff/*": ["./*"] })

And now that I think about it, another potential benefit of merge properties would be it encourages developers to split their extended tsconfigs into isolated configurations. I.e. with template variables, you could selectively pull out the parts of a tsconfig that you want, while with merge properties, you are merging an entire property from all of your extended tsconfigs, so you need to make sure you only have exactly what you need in the extended tsconfigs.

Side note: I would think template variables would be more comprehensive (cover more use cases more completely), while merge properties would be more concise (require less input from the developer)

ijxy commented 7 months ago

@saltman424

Now you could take your point more generically, that mono-repos are outside the scope of TypeScript. I.e. the burden of making TypeScript useful in mono-repos is entirely on npm, Nx, or whoever else wants to take it on.

Yes, I would generally lean towards that as a guiding principle.

In fact, Nx effectively does not intrude on the actual mechanics of sharing code in your project by allowing you to either use npm workspaces (package-based) or TypeScript paths (integrated). In other words, they seem to be treating it as if npm and TypeScript provide two different, valid ways for sharing code in a mono-repo, and they just try to support both.

But npm workspaces and TypeScript paths are not fundamentally not equivalent. npm workspaces are a dependency management solution; TypeScript is a DSL and a transpiler.

I am very skeptical about TypeScript adding complexity to handle problems that are complex, optional and can be resolved in user-land without any changes to TypeScript.

In the Nx case:

tsconfigs are already not WYSIWYG

I said pretty much WYSIWYG. Every property is either the value in your tsconfig, or the value in the config it extends from. Even in a "Russian doll" scenario, you still end up on a single property (or none)–you do not need to mentally merge all the layers in your head to understand what your actual config is.

If extending is a foot-gun, merging is a foot-bomb.

saltman424 commented 7 months ago

@jxdp

But npm workspaces and TypeScript paths are not fundamentally not equivalent. npm workspaces are a dependency management solution; TypeScript is a DSL and a transpiler.

I totally agree. TypeScript paths are not meant to manage dependencies. They make that quite explicit in the documentation. However, it is a valid way to tell TypeScript that you are using another tool to resolve a given path, so you would like TypeScript to understand that path as well. When working with a bundler or executor, this means you can split your code into distinct folders without having them be separate packages, but instead all as input folders for bundling (i.e. every folder will be subject to the same compiler options; have the same dependencies available to them; and be built/transpiled together).

So I agree, these are not interchangeable solutions. They are two different approaches to mono-repos. But they are both viable approaches.

With that being said, this does make me realize that it is not Nx, but really bundlers, like esbuild, and executors, like tsx, that makes this approach to mono-repos viable. The problem, I would argue, is it definitely is not the scope of bundlers or executors to create/manage tsconfigs. They simply consume the tsconfigs for bundling or executing. So I think TypeScript is the most natural place to add this functionality, as numerous bundlers and executors seem to already rely on tsconfigs for specifying this type of path resolution. However, I hear the argument that it was not really the original purpose of paths to become this sort of configuration option for third-party tooling. I just think whether it was intentional or not, this is where we are, so why not make it a little more user friendly. And regardless, paths is not the only property that could benefit from merging and mono-repos aren't even the only case where merging would be helpful.

I said pretty much WYSIWYG. Every property is either the value in your tsconfig, or the value in the config it extends from. Even in a "Russian doll" scenario, you still end up on a single property (or none)–you do not need to mentally merge all the layers in your head to understand what your actual config is.

I think this depends on the person. I find the hopping around much more burdensome than having to append two lists or merge two objects in my head, but others may experience the reverse.

I also see a fair amount of confusion from a lack of merge functionality. E.g. developers confused why including a folder excludes another folder (because include overwrites the extended include).

Also, with template variables, it is all explicit, so I don't think that adds a lot of cognitive complexity, at least for a reader. And we already have merge logic on compilerOptions, so I think even with magic merge properties, people seem to be at least somewhat capable of understanding how it works without too much mental burden.

mrdeveloper-ir commented 6 months ago

is there any fix here ?

silverwind commented 5 months ago

Imho typescript would be better off just supporting typescript.config.{js,ts}, then you can offload all this merge/extend/path logic to the user doing it in their code. Numerous tools already support *.config.{js,ts} files.

Past related issues: https://github.com/microsoft/TypeScript/issues/30400 https://github.com/microsoft/TypeScript/issues/39441

Knagis commented 5 months ago

I just added a patch for tsc.js and typescript.js in our monorepo, to make sure that extends always include the output directory, which in 5.5 is now defined in the root config. I checked and we have ~140 tsconfig files where we defined exclude that doesn't explicitly exclude outDir. Being able to merge the extends from root would have mitigated need for such patch.

As for the arguments mentioned above about using package dependencies - we have a rather large monorepo with hundreds of typescript projects cross referencing each other and we long time ago made a very explicit decision that we only ever want to depend on TypeScript project references and paths to resolve dependencies, we do not want to use the dependencies in package.json files so that we have much better guarantees about consistency and we don't want to be locked into whatever yarn/npm/smth else things how monorepo dependencies in package.json should work.

IMO, option 8 is the better, simpler option. If i would need more granular support, i might as well create multiple base configs.

A question that i didn't see mentioned above.. Is the merge:true (or one of the variations) inherited?

tsconfig.base.base.json
{
merge: true,
include: ["foo"]
}

tsconfig.base.json
{
extends: tsconfig.base.base.json
merge: undefined, // or what if this is false?
include: ["bar"]
}

tsconfig.json
{
extends: tsconfig.json
include: ["baz"]
}

what would be the end result?

hichemfantar commented 4 months ago

My suggestion is to change the current behavior of extending compilerOptions by default instead of overriding them (could be considered a breaking change, may require a major release). And I suggest a new propertry "override" that allows setting which properties will be overriden. the overrideproperty can be defined for each level to allow maximum flexibility.

{
  "extends": "@acme/tsconfig",
  "compilerOptions": {
    "override": ["types"],
    "baseUrl": ".",
    "types": [
      "node",
    ]
  },
}

resulting compilerOptions when "override": ["types"] and "@acme/module-type-aliases" is defined in the original acme config "@acme/tsconfig":

{
  "compilerOptions": {
    "baseUrl": ".",
    "types": ["node"]
  },
}

resulting compilerOptions when "override": [] and "@acme/module-type-aliases" is defined in the original acme config "@acme/tsconfig":

{
  "compilerOptions": {
    "baseUrl": ".",
    "types": ["node", "@acme/module-type-aliases"]
  },
}
PatrykKuniczak commented 1 month ago

This is bizarre, in 2024 this feature isn't implemented.

I want to use it for jsx, baseUrl, types but i can't.

This should work like deepmerge

ciandt-crodrigues commented 1 week ago

Is there any workaround for this to reference the extended config on the project paths?

ciandt-crodrigues commented 1 week ago

To be honest, having a tsconfig.ts where we could freely compose all the strong type configuration file fields (even dynamically if we want) would be much better than these merge options