[turborepo] Setting `dotEnv`-field used file instead of defined environment variables

[weyert]

What version of Turborepo are you using?

1.10.0

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Describe the Bug

I have defined .env.example as the value for dotEnv in turbo.json but it not using the environment variables defined in this file but somehow uses the file itself.

Expected Behavior

Ability to quickly source tens of environment variables by linking up a .env file (e.g. .env.example) via turbo.json

To Reproduce

Create .env.example file

AUTH_URL=
ACCOUNT_URL=
DATABASE_URL=
MAX_REFRESH_INTERVAL=

add it to turbo.json like:

"dotEnv": `.env.example`

Run dry-run on the project, e.g. turbo run build --dry-run=json

Expect the dry-run to list the four environment variables as part of:

"environmentVariables": {
        "configured": [],
        "inferred": [],
        "global": [
          "DATA=",
          "VERCEL_ANALYTICS_ID="
        ],
        "passthrough": null,
        "globalPassthrough": null
      }

Reproduction Repo

No response

[gsoltis]

Hey @weyert the new feature doesn't load the .env file, it just incorporates it into task hashes, since they are sometimes .gitignore'd and thus changes can cause accidental cache hits. Docs coming soon (#5078).

However, I'd love to hear your use case for using Turborepo to load your env vars. What were you doing before, and is there something you would want turbo to do to help?

[weyert]

Ah okay then I misunderstood it. Yeah, my idea was to make it quicker and easy to maintain environment variables listed in turbo.json. Guess, I can write a script that reads dotenv file and update turbo.json manually.

[gsoltis]

@weyert you don't need to enumerate any env vars that are listed in .env files. As long as the .env files are listed for the correct task, we will take them into account in the file hash. You can continue loading them however you like. With this change, turbo won't need them to be loaded beforehand, since the contents of the file itself get hashed.

What will still need to be enumerated in turbo.json are any env vars that are not loaded from a file, but come from some external source. So if you are currently using dotenv my.env && turbo my-task you can probably drop the dotenv invocation, as long as something in my-task loads my.env for you.

[weyert]

Yes, but wouldn't the environment variables in CI affect the remote cache? Especially, when you don't use dotenv files in those environments, e.g. CI or docker builds? (aside, I have to admit I haven't got caching work in those places yet.)

[gsoltis]

If you are running different tasks in CI and locally, then you can configure them separately. If you are running the same task locally and in CI with different env vars, you want those tasks to be cached independently.

[tknickman]

I believe the gap here is that most people use .env files only in dev. They might have env vars X, Y, and Z set to dev values, and stored in an .env file that is not committed. When deploying to production, the production values for X, Y, and Z are set directly on the build process, and not added to any .env-* file.

For that common use case, the dotEnv field in turbo.json won't be helpful, as you would still need to enumerate your env vars as part of env or globalEnv in turbo.json to ensure they are correctly accounted for in the production environments hash.

[gsoltis]

Closing the loop (and issue): with the dotEnv setting, you will catch changes to the .gitignore'd .env file that previously turbo would have missed.

Any environment variables that can be set externally continue to need to be specified in env.