Disable loading .env files

dschnare commented 1 year ago

Description

Be able to opt out of loading .env files.

Motivation

When attempting to migrate our monorepo to use Nx, we have encountered a challenge where our existing .env files are causing tests to fail because they are loaded when we originally didn't intend them to load during tests.

Suggested Implementation

If an environment variable is present like NX_LOAD_DOT_ENV_FILES and set to "false" then no .env files will be loaded.

Another possibility is to check if an env file has been ignored in the .nxignore file. This would allow removing all or only a few env files from the load hierarchy.

dschnare commented 1 year ago

I tried investigating the source, and the runOne() function (defined in the command-line namespace) seems to have a extraOptions.loadDotEnvFiles: boolean it accepts, but I tried hardcoding it to false and the env files are still loaded when running a command like nx run my-proj:test.

dschnare commented 1 year ago

Narrowed it down to two methods ForkedProcessTaskRunner#getDotenvVariablesForForkedProcess and ForkedProcessTaskRunner#getDotenvVariablesForTask. However, I tried hardcoding those methods to each return {} and the root .env file is still loaded.

My test so far has been to install Nx locally and run a dummy command like ./node_modules/.bin/nx run dummy:jump.

Project.json:

{
  "name": "dummy",
  "targets": {
    "jump": {
      "executor": "nx:run-commands",
      "options": {
        "commands": ["echo $MY_VAR"]
      }
    }
  }
}

Root level .env:

MY_VAR=hello

What I am after in this test is when echoing I should see nothing, not the env var loaded from the .env file.

dschnare commented 1 year ago

Got it. PR incoming.

mattrunyon commented 1 year ago

+1. This completely breaks Vite's env setup since Vite doesn't override existing env variables. So nx loads the default .env and then when Vite goes to load the mode specific and local .env.mode and .env.mode.local files, it doesn't overwrite the base since nx has already loaded something and put it in process.env. I even tried @nrwl/vite and it still doesn't work properly.

The only solution for now I think is to rename my start script to development and even then it would only load .env.development and not .env.development.local

jaysoo commented 1 year ago

I think what we might want to do is add an option in nx.json (e.g. loadDotentFile), and when generating a Vite app we can set it to false.

That way, users generating a Vite project will have this set and handled for them automatically. The environment variable can still be added and documented here https://nx.dev/reference/environment-variables#environment-variables for users who may need the behavior for some projects in their workspace.

mattrunyon commented 1 year ago

Another option I could think of is nx only loading NX_ env variables. It seems to be loading all which is causing the conflict w/ Vite.

I think the current setup would also conflict w/ Create React App, but seems that's getting left behind so not a big deal there

jaysoo commented 1 year ago

Loading a subset of .env file isn't supported by dotenv AFAIK.

AgentEnder commented 1 year ago

Loading only a subset would also be a pretty major breaking change when compared to the functionality thats been around for a long time now

edikdeisling commented 1 year ago

are there any workarounds to avoid nx loading .env into global environments? While using Vite this behavior makes .env take into first place before .env.test

lucasvieirasilva commented 1 year ago

+1, this even breaks one of the Nx recipes to build your own executor to compose executors: https://nx.dev/plugins/recipes/compose-executors because when the executor is executed also loads the env vars for .env and .env.local files.

mpacholec commented 1 year ago

are there any workarounds to avoid nx loading .env into global environments? While using Vite this behavior makes .env take into first place before .env.test

AFAIK you can't avoid loading .env by nx yet but you can undo what nx loaded (before running vite) by

unset ${!VITE_*}

apocaliss92 commented 1 year ago

Hi! I just switched from vite -> nx/vite and environment variables loading is quiet broken, loading order is inversed and some files are not loaded at all (.env.development.local). Another problem is that variables replacement does not work anymore (dotenv extended is probably not used)

Is there some quick temporary solution to run on the vite.config.js file? Maybe resetting the env before loading the new one? Thanks!

KarnerTh commented 1 year ago

This seems to be fixed, but it is not well documented https://github.com/nrwl/nx/pull/12602

edikdeisling commented 1 year ago

NX_LOAD_DOT_ENV_FILES not work for me nx affected still loads the .env file even with NX_LOAD_DOT_ENV_FILES set "false". Maybe because true is hardcoded there https://github.com/nrwl/nx/blob/d5f87f79bda0ff1efdf97639c83e1d25a9e265eb/packages/nx/src/command-line/affected/affected.ts#L115

The only option I found is to patch nx with pnpm and replace all loadDotEnvFiles: true with loadDotEnvFiles: false

jantoodre commented 1 year ago

What the current state looks like:

NX_LOAD_DOT_ENV_FILES defines whether the various .env files are read source
NX_LOAD_DOT_ENV_FILES is set depending on extraOptions.loadDotEnvFiles property source
extraOptions is param, where loadDotEnvFiles defaults to true in run-one, run-many, affected
extraOptions are never passed to the methods runOne and runMany
Since the params are not provided, defaults are used ({ excludeTaskDependencies: false, loadDotEnvFiles: true }), overriding the provided NX_LOAD_DOT_ENV_FILES flag.
In Lerna the --load-env-files='false' cli options works because it passes the extraOptions for runOne, runMany (related topics: lerna issue #3371, lerna pull #3375)
In discord one result popped up regarding NX_LOAD_DOT_ENV_FILES, but I'm unsure what to do with that, as I'm fairly new to nx

Essentially it seems that the reason for the flag not working is because where this flag should be provided, it is not.

When this issue gets fixed, this flag should be described in the docs

Currently resolved it for myself similarly to @edikdeisling, where I replaced the default loadDotEnvFiles default values to false in my node_modules nx/src/command-line/run-one/run-one.js and nx/src/command-line/run-many/run-many.js. But this is guaranteed to break after re-installing deps / upgrading nx version

sergioflores-j commented 8 months ago

Any news regarding this issue? Will it be worked on after NX 18 (crystal) has been launched?

I recently migrated a monorepo with a few Serverless (Framework) projects. They rely on the .env.${STAGE} resolution from it (https://www.serverless.com/framework/docs/environment-variables) but these projects also contain .env (with no stage) for fallback stages. (e.g. ephemeral dev environments).

When running the deploy command via e.g. "nx deploy" it would never succeed because .env is always there already.

What I did (not future proof as mentioned before) is patching loadDotEnvFiles as follows:

--- a/bin/nx.js
+++ b/bin/nx.js
@@ -88,6 +88,8 @@ function main() {
  * - .env.local
  */
 function loadDotEnvFiles() {
+    if (process.env.NX_LOAD_DOT_ENV_FILES !== 'true') return;
+    
     for (const file of ['.local.env', '.env.local', '.env']) {
         const myEnv = (0, dotenv_1.config)({
             path: file,

--- a/src/tasks-runner/run-command.js
+++ b/src/tasks-runner/run-command.js
@@ -93,7 +93,7 @@ async function runCommand(projectsToRun, projectGraph, { nxJson }, nxArgs, overr
             lifeCycle,
             nxJson,
             nxArgs,
-            loadDotEnvFiles: extraOptions.loadDotEnvFiles,
+            loadDotEnvFiles: false,
             initiatingProject,
         });
         await renderIsDone;
@@ -112,9 +112,9 @@ function setEnvVarsBasedOnArgs(nxArgs, loadDotEnvFiles) {
     if (nxArgs.outputStyle == 'stream-without-prefixes') {
         process.env.NX_STREAM_OUTPUT = 'true';
     }
-    if (loadDotEnvFiles) {
-        process.env.NX_LOAD_DOT_ENV_FILES = 'true';
-    }
+    // if (loadDotEnvFiles) {
+    //     process.env.NX_LOAD_DOT_ENV_FILES = 'true';
+    // }
 }
 async function invokeTasksRunner({ tasks, projectGraph, taskGraph, lifeCycle, nxJson, nxArgs, loadDotEnvFiles, initiatingProject, }) {
     setEnvVarsBasedOnArgs(nxArgs, loadDotEnvFiles);

NX_LOAD_DOT_ENV_FILES could be an option, but setEnvVarsBasedOnArgs always sets it to true and extraOptions is not available to configure anywhere.

Gnucki commented 5 months ago

The env variable NX_LOAD_DOT_ENV_FILES should be functional since version 19.0.4 : https://github.com/nrwl/nx/releases/tag/19.0.4

Thank you @xiongemi !

leoselig commented 5 months ago

@Gnucki While this is a great step forward regarding the issue, I believe it's not sufficient yet. I'm using the nx VS Code plugin to run tasks. There isn't an option to provide default env vars this way to nx to pick up. I would propose this option living in the nx.json or the project.json. I think this is preferable since nx not loading env files seems to be either always desired or never, not just for some workflows. An alternative could be to have the extension either exposing a setting for supplying env vars to nx (seems unnecessarily generic imho) or providing the loadDotEnvFiles as an option in the extension. Again, I think the first solution makes more sense in terms of the original use case.

Gnucki commented 5 months ago

@leoselig You're probably right. This is where we expect it to be at first thought.

After some investigations, it seems that the current implementation is loading env variables from .env* files, then unload them if NX_LOAD_DOT_ENV_FILES is false. I didn't test more further, but this design may lead to erase variables you specified if they exist in your .env* files. I didn't try too long, but the forward of the variable to the nx child processes during a run-many seems deficient too.

iambpn commented 1 month ago

Are there any updates on this? What's stopping this feature to be added. This seems like a simple fix.

xiongemi commented 2 weeks ago

when setting NX_LOAD_DOT_ENV_FILES to false, it should not load env files.

xiongemi commented 2 weeks ago

you can specific this env NX_LOAD_DOT_ENV_FILES in your root env files. If you set it as false, and nx will get picked up this value, but later it will unload values in your root env files for task specific envs. here is code where root env files got picked up: https://github.com/nrwl/nx/blob/e345bc7b11e55b21ffef1f3b680582c53b1a1309/packages/nx/src/utils/dotenv.ts#L12

here is code where it got unloaded: https://github.com/nrwl/nx/blob/e345bc7b11e55b21ffef1f3b680582c53b1a1309/packages/nx/src/tasks-runner/task-env.ts#L230

You can still specific this env NX_LOAD_DOT_ENV_FILES=false in files like '.env', '.local.env', '.env.local' in your workspace root and it should be picked up.

leoselig commented 2 weeks ago

you can specific this env NX_LOAD_DOT_ENV_FILES in your root env files. If you set it as false, and nx will get picked up this value, but later it will unload values in your root env files for task specific envs. here is code where root env files got picked up: https://github.com/nrwl/nx/blob/e345bc7b11e55b21ffef1f3b680582c53b1a1309/packages/nx/src/utils/dotenv.ts#L12

here is code where it got unloaded: https://github.com/nrwl/nx/blob/e345bc7b11e55b21ffef1f3b680582c53b1a1309/packages/nx/src/tasks-runner/task-env.ts#L230

You can still specific this env NX_LOAD_DOT_ENV_FILES=false in files like '.env', '.local.env', '.env.local' in your workspace root and it should be picked up.

@xiongemi This is known but not sufficient. See my comment here: https://github.com/nrwl/nx/issues/14010#issuecomment-2142275003 My take on this is is that env loading behavior applies to such an early stage in the process that it must be controllable by ALL components that actually spawn such processes. The VS Code extension for instance is an official one of them, so it needs to be considered here.

nrwl / nx