Another issue asking about multiple env files

[datner]

I, and the rest of the ecosystem, use, understand, and support multiple env files. This includes any modern tooling I can think of like vite.

Despite production vs development (or env vs env) being the poster-child example, there are many other differences that can pop up within the same environment dev vs test local vs proxied live vs mocked debug vs prod-like

I've read that it's not recommended, so I've looked through the repo, this is a common question with confounding answers for some reason. The suggested solutions I could find are as such:

instead of doing that, dont-do-that

not-doing-that will align with the recommendation, but it's not solving the issue that required the multiple envs to begin-with!

add some env var to indicate [stateA] or [stateB]

this just pushes the env var logic and content inside the application, which is the pattern that we wanted to avoid to begin with

and put [STATE_A]_VAR and [STATE_B]_VAR in the env

even ignoring how convoluted this is, it also just breaks a different principle that states that env vars are independent and orthogonal to each other and adds noise to the env with both being supplied

ssh into different machines and manually create specialized .env for each.

Would work fine, if a bit cumbersome, for bare metal or VPS. But I know I don't need to tell you that these are no longer the norm if to put it lightly and even those who do use these low-level infrastructure nowadays have everything automated in a ci/cd pipelines. Sometimes devs don't even have ssh access and only devops do. We can debate forever about how bad that is but me agreeing does not change the fact that this is common in modern development departments

make a config folder and add [stateA].env and [stateB].env and by adding STATE=A you can then check which file to load dotenv(process.env.STATE === "A" ? "./config/a.env" : "./config/b.env"

This is just multiple .env with extra steps

use .env.vault

I respect the hustle, and this is the best answer to passive-aggressively tell someone that you've worked for free for long enough. A legit and very well-earned response(!!!!!). But in this case this is not a real solution when the alternative is just adding another .env.proxy along-side the .env and the deterrent from doing that is "we recommend you don't"

And usually, after these answers, some randoms join in advertising their tooling that does allow and even recommends multiple env files, their setups, their hacks and nifty scripts, and other such things. This signals to me a very clear "We appreciate the theory, but heres how to solve your issue in practice"

So I will ask again, and hopefully for the last time, how are we supposed to handle multiple revisions on the same machine? If the answer is "for this example its fine to use multiple env files" thats fine, but should be communicated more clearly

Concrete example: I have a monorepo with webapp and api. This scenario is from the POV of the webapp. Usually I want to just talk to the prod server, but sometimes I have a cross-cutting concern and I want my webapp to talk to the local api instead. I am also sometimes interested in just mocking the api. When I'm testing, it's also not cut and dry. Complex flows are being developed while being automated, but still ofc conditions are different in tests. So I need both the test conditions, and the ability to override params easily for the aforementioned flow. This all is on the same env. So theoretically, I will have these .env files (none of which are committed to git. Everything is ignored)

.env
.local.env
.mock.env
.test.env

But I want to use my tools correctly, so I need to eliminate everything besides .env, and on the other hand, I don't want something convoluted or contrived. I pitched .env.vault to my team lead but he placed a hard no because he is too invested in azure key store or something and won't listen to reason. What do I do?

[motdotla]

Great thoughts and strong exhaustive points @datner.

You're not the only one that has pointed that out about .env.vault.

Have you seen or tried dotenvx?

https://github.com/dotenvx/dotenvx

It has a mechanism that allows you to manage multiple .env files by doing something like this:

dotenvx run --env-file=.env.local --env-file=.env -- node index.js

docs

Would that solve your problem?

(dotenvx is taking all the learnings from dotenv and dotenv-vault and trying to wrap them up in a solution that runs everywere and works everywhere. You can use it with .env.vault or without. Your choice. It's a more flexible tool so that you can mold around your own development flows. It certainly does still recommend using .env.vault for production but you don't have to if your team lead is against that now. Admittedly, it is a modern approach that will take time for the community to adopt.)

[motdotla]

see the npm example in the dotenvx README [0] for a drop-in replacement to dotenv. (it still uses dotenv under the hood for parsing of your .env files so no behavior change/risk. [1])

[1] https://github.com/dotenvx/dotenvx/blob/main/src/lib/main.js#L5

[datner]

Thank you for addressing my issue @motdotla ! dotenvx, and all the other aforementioned tools, setups, scripts, and hacks, would all work just fine. dotenvx would just be the undisputed best option ofc 😉 . Yes supporting multiple env files was never an issue, even with its quirks. The question was more about the bridging between the theory and praxis.

In a way, the answer can be read as a combined

use .env.vault

and

for this example its fine to use multiple env files

For the former I would try to give it another go, I do believe in this tool. The latter I want to go a bit more in-depth into

dotenvx being the next iteration and supporting multiple env files (representing multiple envs) ootb is a foundational shift of ethos from Twelve Factors, going from this to this. I of course could not care less, this is not my project nor do I get money from people abstractly following principles from Twelve Factors. But I'm curious why? Is it because of practical concerns, with theory having a hard time manifesting praxis? Or is this a concession with the status quo? Or just a change of mind/understanding? All legitimate ofc.

Regardless, I'll be migrating to dotenvx while waiting your answer

P.S closing the issue while I'm writing a response, rude 😠

[motdotla]

Is it because of practical concerns, with theory having a hard time manifesting praxis? Or is this a concession with the status quo? Or just a change of mind/understanding? All legitimate ofc.

A little of all of that.

1. Practical - it is simpler to maintain a .env and .env.production file. The symmetry makes it easier to make sure you haven't misconfigured your production environment. Too many times I and others have missed an environment file in prod because the UI/config tool is different than the elegant .env file.
2. Concession - very popular frameworks like Next.js have popularized multiple env files particularly around the .env.local mechanism. There is a good use case for this especially as 3rd party cloud services became ubiquitous (which was not the case when .env was first created)
3. Change of mind - the introduction of the .env.vault file makes it much more elegant to manage all your environment variables in .env.environment files and then encrypt them to a single file and single decryption key on your server. This greatly helps minimize the security risk mentioned in number 1

We learned a lot over the last 10 years as a community. I think acknowledging these needs and introducing the .env.vault file will give the dotenv pattern at least another decade of heavy usage. It is more important than ever because the other patterns around secret and config management out there are largely proprietary or require keeping a service to be running. I don't like that future so dotenv will continue to carve out the more elegant and accessible solution.

P.S. closing the issue while I'm writing a response, rude 😠

haha. touché

[datner]

These are very astute observations, except one:

give the dotenv pattern at least another decade of heavy usage

I think it's very pessimistic to assume that technology will change in such a fundamental way that env vars would cease to be relevant in the next 10y, and the dotenv pattern long surpassed methodology and 'ascended' to foundational concept. I'm sure you've heard such sentiments before, but a perfect companion to dotenv (conceptually) is direnv. There is already a lot of overlap, especially for development environments where env vars need to exists statically. for example rusts env! macro.

If dotenv is exploring new horizons with dotenvx and env.vault, maybe a deeper integration with it could be a worthwhile examination?

[motdotla]

well a lot can change in 10 years but I do agree with you. I was being conservative rather than pessimistic.

If dotenv is exploring new horizons with dotenvx and env.vault, maybe a deeper integration with it could be a worthwhile examination?

exactly. i think we will get there. i think it is our job mainly now to build good tooling around the .env.vault file. it's a very elegant solution to a hard problem. as tooling and education on it gets better I think tools like direnv could also adopt it. but in the spirit of open source i think it best we focus on our task and allow other tools to play things out as they wish (or as devs request it from other tools). i think it's best for the community and the industry when it happens that way.