the creeping scourge of tooling config files in project root directories

[boneskull]

Splitting this out from #71 which was rather broad. There are (at least) two separate issues there; one is where user-specific config files should live, and another is where project-specific config files should live. This issue is about project-specific config files.

The files we're talking about are configuration files for development dependencies and are committed to VCS.

Many projects suffer from the problem of too many config files in the project root, as @iansu tweeted:

We put config files in the project root because we put config files in the project root. There is no reason other than a lack of an alternative convention.

We can do better than this. The idea is we aim for a "critical mass" of popular tooling authors moving their project-specific configuration to a subdirectory (e.g., .config/). If we can agree on a subdirectory, and change our tools to support the new subdirectory, we will significantly reduce this problem. It's our hope (though not a strict requirement) that the convention we agree upon will be flexible enough to reach beyond the JS ecosystem. For instance, your .travis.yml and netlify.toml could live in this directory as well, if this idea becomes popular enough.

We should get buy-in from maintainers of popular tools; if these tools adopt the new subdirectory, it's likely the ecosystem will slowly follow suit.

cc @nodejs/tooling @nodejs/package-maintenance

[styfle]

Would this new subdirectory also include package.json config?

[boneskull]

@styfle Whether any given tool uses the subdirectory is up to the maintainers of that tool. For npm in particular, this seems like perhaps a risky move, given there are many assumptions across userland tools about where package.json lives.

[boneskull]

(It's not really npm's to move, either... they aren't the only consumers of it, obv)

[coreyfarrell]

Moving package.json to a subdirectory would also be problematic for Node.js where the package.json "type" field is set.

[ljharb]

Moving eslint config would be problematic for the same reason, since it composes up the hierarchy, and describes the current folder.

[bcoe]

what if a folder such as .config/ had identical semantics to if the file existed in the root? e.g., test/.eslintconfg.json still took precedence.

[ljharb]

Then how would I lint my JS config files inside .config?

[boneskull]

I don’t think that’s an unsolvable problem.

[ljharb]

It’s a problem that doesn’t need to exist tho. I think it might make sense to come up with a conventional location for some kinds of config files - project-level ones, mainly - but not for directory-level ones, like eslint/babel/npmrc/gitattributes perhaps/etc.

[jedwards1211]

@ljharb well .config/.eslintrc would apply to JS files inside .config since it applies to the parent folder. There would only be an issue if you JS files in .config and parent folder to be linted differently.

@boneskull on the other hand, IDEs could be made to show dotfiles in a pseudo-directory.

I agree that this becomes a bit disorienting, especially when I have a bunch of project folders open in my workspace and I'm scrolling between them.

[ljharb]

Yes, i would assume my config file would match the node version eslint runs in, but my actual files would match the node versions my production code targets - iow i think that’s the most common case.

[ahmadnassri]

this issue mentions "user-specific" and "project-specific" config files, I would argue, while those are important and the noise ratio is quite high per project, however a more important vector to consider is "org-specific" config files.

regardless of team size, small and large teams might be in a situation where they have to manage hundreds of projects (up to thousands in case of enterprises), those projects might be independent repositories or a collection of mono-repos with many packages and/components within.

a potentially more inclusive approach to consider here, is having a central external spot for all config files, that apply by default to each project / repo, and any local file is considered inheritance / override, thus the Patten becomes:

• no local configs => use org-wide defaults
• some local configs => use org-wide defaults and override specific declarations

this would massively shrink the configuration footprint, for both individuals and teams.

some challenges with this approach:

• how do you make clear to a visitor that this projects is pulling it's config files from somewhere else?
• would local files "override" or "extend" the source

some good relevant references here (though still rely on a local file)

• eslint shareable configs
• semantic-release shareable config
• probot "extends"

[ljharb]

a config tho that lives outside the repo won’t be easily shareable across multiple devs.

[ahmadnassri]

a config tho that lives outside the repo won’t be easily shareable across multiple devs.

I would argue eslint, semantic-release, renovate bot and others have done a good, positive job of setting a pattern here (they use packages) that make it easily shareable

[ljharb]

Totally! But those all use per-directory config files, even if they reference a shared config. You’d still need a way to target an individual directory with its own config, which is what eslint config flies already achieve. Moving them into another folder adds complexity because that folder needs its own settings too.

[ahmadnassri]

yep, agreed. I don't have a good answer / follow up to be honest. I wanted to at least have a mention of a 3rd "vector" beyond user and project to be included in case someone has some further patterns to surface

I've solved this problem in the past by using Docker as the build tool, a central "org-wide" image (named "build-essentials") that hosts all the actual dev dependencies (webpack, eslint, etc..) with a single Dockerfile per repo that it inherits "FROM build-essentials", repos optionally have overriding local configs, but the defaults are all includdd in the build-essentials image.

I also realize "use docker image dependencies" is not necessarily the approach sought here, but worth a mention as a reference how I tackled this in the past for a very large team (~400+ humans, ~800+ projects)

[shannonmoeller]

If someone wants to add support for this pattern to their module, I created shannonmoeller/find-config back in 2015 for exactly this reason. Glad to see this idea being championed agin.

[boneskull]

fwiw, with eslint, I avoid putting eslintrc files in multiple directories, and instead use the overrides section in a single root config file to control directory-specific configuration. I would not use multiple config files unless that root config became massive.

[wesleytodd]

I think a great next step would be to chat with @nzakas and the other ESLint maintainers, as that would be one of the most widely impactful projects to support this. Maybe also @hzoo and the other Babel maintainers.

[jkrems]

I think a potential alternate perspective: Putting all those files config files into ./.config seems a bit like sweeping the problem under the rug. There's still 10s of config files with potentially overlapping settings that may need to be updated and/or referenced while trouble-shooting. They are just a bit harder to find now. To me a more interesting problem would be - could the number of those files be reduced somehow?

[wesleytodd]

hey are just a bit harder to find now.

If you take a look at many projects, these files are rarely changed. It is very common for you to see 10 files at the root of a project which haven't been changed in 4 years, when the lib or src directory was changed last week. IMO, this is a great reason to make them harder to find.

could the number of those files be reduced somehow?

It can, use less tools. I kid a bit, but seriously the number of config files is IMO a "smell". There are some reasons which are valid and bring enough value to a project to make it worth it, but shifting complexity from a number of files into less files most likely will not remove the complexity. The only way is to actually remove the complexity by removing the tools or some feature of those tools.

I would argue eslint, semantic-release, renovate bot and others have done a good, positive job of setting a pattern here (they use packages) that make it easily shareable

This is one great point, if you can make great defaults or easily shareable configs, this problem is reduced. This is basically the only reason I use standard, it bundles up the config file into a package install, moving the complexity from my source code into it's source code. It also might be a solution to @jkrems point.

[jedwards1211]

@jkrems

could the number of those files be reduced somehow?

Personally I like putting config in package.json for tools that support it. Some people don't like those extra bytes winding up in the published package though so it would be nice if npm had a built in way to exclude some keys in package.json from publishing.

Or maybe if you use something like https://github.com/romefrontend/rome, but it's hard to imagine Rome will be able to beat all these other specialty tools at their own game.

[jedwards1211]

I get the impression half the reason Sindre Sorhus made xo is just so that he could have the linter defaults he wants without any extra config in his repos.

[bnb]

Personally I like putting config in package.json for tools that support it.

Quite honestly, a solution I'd prefer to see: instead of trying to force a best practice on the ecosystem, support defining this in package.json. Have a configs property (using the name previously shared) that either has a string value of a path that configs can be found in or is an object whose properties are the name of the config (i.e. babel) and the value is the relative path to the config. Tooling could read read that and let people shove them wherever they want - in a .configs directory or anywhere else.

[ljharb]

Another direction to pursue, as opposed to trying to change "there's config files in the root" (since there are often good reasons for it), would be asking github if there's a way to hide those dotted config files on first view of the repo page (with a "show config files" button, or something).

[shannonmoeller]

Toggling visibility of dotfiles on github is a bad idea as it would be a convenient place to hide malicious code. It also doesn't catch files like *.config.js or config.*.js.

[Eomm]

Toggling visibility of dotfiles on github is a bad idea as it would be a convenient place to hide malicious code. It also doesn't catch files like *.config.js or config.*.js.

I agree.

If the target is to ease the project navigation I'm not sure that suggesting a new plugin/tool to be installed in the IDE would be the solution.

I would use the directories object that npm is already supporting. We could add a conf key that defines the root path to the config files.

Tools should read that path (if any) to get the config.

Regarding

You’d still need a way to target an individual directory with its own config, which is what eslint config flies already achieve

I think that all remain as is: the user will add that file in the right dir this setup will change only the load of the "root" config, without changing the cwd.

those projects might be independent repositories or a collection of mono-repos with many packages and/components within.

For this approach, instead, I can think only to a new lerna command.. not sure how to deal with this use case since there could be many conflicts if those components use different major versions (thanks to legacy code of course 😀)

[nzakas]

At least once a year someone asks ESLint to move our config files into a new directory to avoid "too many config files in the project root." Each year we say the same thing: as long as you don't pepper your project with .eslintrc files, you can move your config file wherever you want and use the -c command line option to read it.

FWIW, we're working on a new config system (https://github.com/eslint/rfcs/tree/master/designs/2019-config-simplification) that will eliminate the hierarchical/cascading config part of the system, but our advice will still be to use the -c flag to specify which config file you want load.

I really think the long-term solution is just for tools to allow end-users to specify where they want their config files to be loaded from and not try to get everyone to agree to store config files in a specific directory.

[qballer]

@nzakas made some valid points with the -c option. To add on top of that I believe config files should be explicit and not implicit. Meaning the default option should be to mention the required config. This can give package authors the time to think where they want to put their config files. This could be annoying in the terminal but solvable via the scripts.

Another somewhat hacky idea is to monkey patch the fs for common config file. I'm unsure what are the proper ascetics of this.

[shannonmoeller]

use the -c command line option

Thank you for supporting this command line flag. It's a great start! But many users make use of GUIs, IDEs, or daemons which don't support setting command-line flags. So the onus is still on users to create bespoke solutions per project. It's disheartening to see when there's a well-documented open standard like XDG (the root .config/ idea) that would fix the problem for thousands of developers if a few dozen tools would agree to support it:

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

If the top 10-15 Node.js projects supported this, support for the pattern would likely ripple out naturally.

[dominykas]

I strongly disagree with the idea that tools should always explicitly take a config option from the CLI and not provide a default. That would result in very poor UX and a lot of extra work for developers using such tooling. I'd like to focus on productive work and making fewer decisions, not more, please.

GUIs, IDEs, or daemons which don't support setting command-line flags

Or even environment variables. Which is what XDG relies on? However it has to also be said that XDG is more geared towards global configs? And it's also Linux only?

I would use the directories object that npm is already supporting.

I'm very much not against this idea, but I suspect that having a smart default would probably gain more adoption?

[varunarora]

I don't intentionally want to play devil's advocate here, but just want to understand what the issue is here.

The fact that the root directory has a lot of config files? And that the directory on a user's computer looks very filled up when they clone or fork it?

I consider myself a deeply design-savvy programmer, but am unable to see the problem here. There are tens of solutions, of course, but please could you help me understand what you consider to be the pain here?

[dds]

We put config files in the project root because we put config files in the project root. There is no reason other than a lack of an alternative convention.

Is the problem with putting them in the root directory of the repo that GitHub's UI is devoting a lot of screen real estate to dotfiles that would be hidden if we cloned the repo locally? Could we petition GitHub to not show files beginning with a . by default, perhaps showing them hidden with a rollover?

[jspears]

I wrote mrbuilder to help solve this issue. Particularly in monorepos. Its kinda like a universal configurator for webpack,eslint,karma and a ton of others. It solves the issue by having 1 file with all the configurations. This is not everyones cup of tea; but its proved useful.

[sneak]

The root is a good place for project-wide settings.

GitHub should hide dotfiles by default, like ls does. They are dotfiles for a good reason. Perhaps other metadata (like package.json) should follow the .git example and also be dotfiles.

[DanielTate]

Would this new subdirectory also include package.json config?

This might cause issues for npm script execution?

[intrnl]

Would this new subdirectory also include package.json config?

Moving package.json to .config folder seem like an odd choice, given that some of its fields are more of a metadata in a sense.

First off, who is the intended target for this new .config folder? Is it only for tooling/development, or is it also a place to put files necessary for runtime?

I suppose given the title this would've meant for tooling/development, so here's what I propose. Keep the necessary fields like name dependencies and post/preinstall scripts on the project root folder, but for the rest like publishConfig or build scripts, move it into package-dev.json

[alganet]

The IETF folks solved this problem on URIs using a .well-known namespace:

RFC about it: https://tools.ietf.org/html/rfc5785 Also: https://en.wikipedia.org/wiki/List_of_/.well-known/_services_offered_by_webservers

[intrnl]

The root is a good place for project-wide settings.

GitHub should hide dotfiles by default, like ls does. They are dotfiles for a good reason. Perhaps other metadata (like package.json) should also be dotfiles.

Hiding config files because they are dotfiles is an odd choice, I think most devs would just turn it off because it hinders their ability to poke through the project configuration from the website, and most beginners would find it confusing because it looks as if their config files didn't make it onto the git repository (not everyone knows Unix conventions).

[lyjia]

Perhaps the problem is too much tooling. Stop building apps requiring so many config files! Config files in the project root is an old and time-honored convention, and this issue seems to be more about aesthetic preference than good practice.

We put config files in the project root because we put config files in the project root. There is no reason other than a lack of an alternative convention.

This is not true. Config files in the project root is an extension of the Linux tradition of config dotfiles in the user's homefolder. Some of the comments on HN (https://news.ycombinator.com/item?id=24066748) are exploring this. Placing configuration in this fashion can potentially grant lots of flexibility if the applications are aware of it, because of the implicit recursion that takes place. Depending on the app, overrides can be placed in subfolders. It is also easy to traverse programmatically, because you can recursively search parent folders until you hit the dotfile you need, and it won't matter what your working directory is as long as its inside the project root path. This is how a number of frameworks (like Rails) search for config data or classes to autoload and it works very well.

[wyaeld]

I don't like the idea of hiding them. What would be good is having a common standard for "configlocations" file, (call it whatever people agree) that just has entries for locating various config files. Then if I want to put all my editorconfig and eslint things under /config, I can. By default nothing changes, but as tools, eg. NPM, offer support for this configlocations file, then I just organise. Opt-in won't break anything, but there is no implicit or assumed behaviour. Then need a critical mass of tooling to add support it.

[shannonmoeller]

As stated previously, hiding files in the github UI is a security issue and is inadvisable.

Saying that you don't see an issue doesn't negate that many others do.

[hussaibi]

Disclaimer: I came here from HN. I've never looked at this repo. I have no business telling people close to the problem what to do. But, I think tunnel-vision is happening.

My understanding of the problem: mandatory hardcoding of configuration files are causing a code smell.

I agree. The code smell is that this is bad aspect oriented programming. Packages should be reserving config prefixes and allow user-defined prefixes to be applied on code arbitrarily. Thus, application aspects should have their own config prefix, all in one file.

The Java Spring Framework does this ^ via Java annotations and application properties/yaml files. Python is now able to do this via Facebook's Hydra library (monkey-patching as was earlier mentioned). Unfortunately, it requires 2 things:

1.JS Decorators (mandatory)

2. JS Class Fields (for better UX)

But all this ^ only address application properties for different environments (i.e. default, local, dev, test, qa, prod). So this would effectively reduce the .js config files down to the # of application environments you're deploying to.

I don't know what the project's ES version being used, but I'll guess it's not using proposal language features. This ^ likely isn't something that can be effectively adopted at this time.

---

If possible, I'd defer taking action until JS Decorators and Fields can be used to refactor the tooling w.r.t its config dependencies.

[kuon]

For compatibility I have some wild idea.

Maybe we could (I mean for projects in general as this issue is going a bit out of scope, came from HN) start using .config (or similar) immediately, and have a tool stow those files to the project root (similar to GNU Stow).

The immediate workflow would be something like:

• create .config/
• move all project config in it
• add .config/ to VCS
• symlink everything in .config/ back to project root ..
• add all those symlinks to .gitignore

Ideally git could have a stow feature that does this automatically.

This is just an idea, I have looked in it in details, but I use gnu stow to manage my dotfiles.

[immanuelfodor]

Maybe it's time to standardize default project folders as Linux has /etc, /usr, /lib, /var, and so on?

It wouldn't need to say where to put JS files or Java source code or PHP files, just the high level architecture of a standard development project. For example, a raw braindump just to get the hang of it:

• /docker
• /config
• /k8s
• /src
• /vendor
• /build
• ...

Maybe these could have their /.../local version in the standard, I don't know. But we could figure it out together and make it standardized.

Edit: Maybe the docker and k8s folders are also too specific like the Java or JS source code examples in the second paragraph. It could go into a cicd folder, or infra for infrastructure.

[sergeykish]

@kuon it is much simpler if you symlink another way

• create .config/

• symlink files in project root to .config, .gitignore them

• hide project root files (nautilus hides .hidden though extended attributes would be better, simpler to patch ls)

This way everyone may configure as way wish - config or .config or docker, k8s by @immanuelfodor, no need to change tools, no suffering by those who likes status quo. And some day there would be no need to decide should the file be hidden upfront (with dot).

[extrowerk]

While i support moving the config files into a subfolder, i am vehemently against using dot-files and dot-folders. This is nixism, and we all know well how big mess this strategy made in our $HOME, so consider use a simply named folder for this, and don't try to cover your back with bs like "Well, nixes doesn't lists those folders, so if we can1t see there is no problem!" because there are operating systems other than *nixes too on planet earth .

[sneak]

Collapsing/hiding dotfiles in the GitHub UI is not a security issue at all, any more than code or configs in a subfolder is; in both cases it's a click away.

[kajmagnus]

Is it necessary to hide the config folder and files — I mean, .config instead of config or conf?

(most people here write .config)

Or would it be nice to see config files by default e.g. if opening in a file system browser — meaning, conf/ or config/ (no dot). I often look at config files and find it mildly annoying that they're . -hidden.

(However, .well-known with dot . makes totally sense — that directory isn't so interesting in itself, for humans; it just tells the not-so-bright computers where to find things. :+1: for .well-known/ )

[timotgl]

What if the tools adopted using environment variables to customize the location of the root config file? Something like

ESLINT_ROOT_CONFIG_PATH=./config/.eslintrc

And the config file would be interpreted as if it was actually located in the root.

So you'd need for example only one .env to get all the rest of the files out of the way. Or just the ones bothering you.