vercel / next.js

The React Framework
https://nextjs.org
MIT License
124.72k stars 26.62k forks source link

Docs: Environment variables documentation implies they're always read at build time #39299

Open richardscarrott opened 2 years ago

richardscarrott commented 2 years ago

What is the improvement or update you wish to see?

https://nextjs.org/docs/basic-features/environment-variables#loading-environment-variables

Note: In order to keep server-only secrets safe, environment variables are evaluated at build time, so only environment variables actually used will be included. This means that process.env is not a standard JavaScript object, so you’re not able to use object destructuring. Environment variables must be referenced as e.g. process.env.PUBLISHABLE_KEY, not const { PUBLISHABLE_KEY } = process.env.

The above mislead me (into using the legacy serverRuntimeConfig / publicRuntimeConfig) as it says environment variables are evaluated at build time which, from my tests and speaking to users on Discord, isn't the case if you're reading them in API Routes or getServerSideProps.

As I understand it, process.env.* works just like any old regular node server does and will read from the global.process.env.* object at the time the code is executed; so for getServerSideProps or API Routes that's runtime and for getStaticProps that's presumably build time?

It's my understanding, the only time next.js will do something fancy with env vars is when code references process.env.NEXT_PUBLIC_* as they'll be inlined into the client bundle at build time? (I guess it also replaces process.env.* with undefined).

Additionally, I've found that destructuring works fine even for vars which are inlined, e.g. const { NEXT_PUBLIC_FOO } = process.env?

Is there any context that might help us understand?

I needed runtime environment variables as our current pipeline builds just a single artefact designed to be deployed to multiple environments.

Does the docs page already exist? Please link to it.

https://nextjs.org/docs/basic-features/environment-variables#loading-environment-variables

DX-1893

natBizitza commented 1 year ago

I have maybe similar issue deploying on Amplify. The API keys that we use in our API routes are undefined. That doesn't happen to me on Vercel. I am not sure yet what is causing the problem.

gudvinga commented 1 year ago

We have the same problem, we needed runtime environment variables as our current pipeline builds just a single artefact designed to be deployed to multiple environments.

We used publicRuntimeConfig, but now it's doesn't work as expected with Output File Tracing because Note: next.config.js is read during next build and serialized into the server.js output file. If the legacy [serverRuntimeConfig or publicRuntimeConfig options](https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration) are being used, the values will be specific to values at build time.

faces-of-eth commented 1 year ago

Also finding this confusing. For instance, I should be able to mount a secret file as .env.local in my dockerfile at runtime to specify server configurations that are different in production and staging. The documentation leads me to think that isn't possible.

dtinth commented 1 year ago

When building for standalone mode (Output File Tracing) the runtime config does get baked into the server bundle at build-time. I stumble upon this problem when building a Docker container and realized that all the secrets have been baked into the image and not configurable at runtime.

As a workaround, to read environment variables at runtime, use global.process.env instead of process.env.

sergiohgz commented 1 year ago

When building for standalone mode (Output File Tracing) the runtime config does get baked into the server bundle at build-time. I stumble upon this problem when building a Docker container and realized that all the secrets have been baked into the image and not configurable at runtime.

As a workaround, to read environment variables at runtime, use global.process.env instead of process.env.

I'm facing this issue some days ago. I create an image that is shared across multiple clients and environments (staging, production), so publicRuntimeConfig and serverRuntimeConfig is really different between containers run with the same service image. I tried replacing with global.process.env in next.config.js, but it doesn't work as expected (using Next 12 and not latest minor available if I remember). Are you using it directly on sources or throught next.config.js, like in this minimal example of my project? Thank you so much all of you

next.config.js

module.exports = {
    // output: 'standalone', // if uncomment, dynamic variables won't load
    serverRuntimeConfig: {
        BACKEND_HOST: process.env.BACKEND_HOST,
    },
    publicRuntimeConfig: {
        ENVIRONMENT: process.env.ENVIRONMENT,
        CLIENT_ID: process.env.CLIENT_ID,
    },
};

.env.production

ENVIRONMENT=staging-client1
BACKEND_HOST=backend.somewhere.example
CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

src/config/public.js

import getConfig from 'next/config';
const { publicRuntimeConfig } = getConfig();

export default {
    environment: publicRuntimeConfig.ENVIRONMENT,
    clientId: publicRuntimeConfig.CLIENT_ID,
};

src/config/server.js

import getConfig from 'next/config';
import publicConfig from './public';

const { serverRuntimeConfig } = getConfig();

export default {
    ...publicConfig,
    backendHost: serverRuntimeConfig.BACKEND_HOST,
};
dtinth commented 1 year ago

@sergiohgz Don’t use serverRuntimeConfig or publicRuntimeConfig as any values set in next.config.js will be baked (“serialized”) into the build at build-time.

I explain it in more details in my notes here — https://notes.dt.in.th/NextRuntimeEnv

sergiohgz commented 1 year ago

@sergiohgz Don’t use serverRuntimeConfig or publicRuntimeConfig as any values set in next.config.js will be baked (“serialized”) into the build at build-time.

  • For serverRuntimeConfig, just read from global.process.env directly in API routes or getServerSideProps.
  • For publicRuntimeConfig, there is no direct alternative right now. Fetch the data you need from server side and pass it to client via getServerSideProps.

I explain it in more details in my notes here — https://notes.dt.in.th/NextRuntimeEnv

Thank you for you fast reply. I’m looking into alternatives and reading your articles, I can inject server variables via environment variables with Kubernetes (loading them with global.process) and create a simple JS file that will be load in client side via script file, injecting values into window (is not the most efficient way, but useful, I use this way in client pure React apps). I will look in other alternatives in the next weeks. Thank you again for your response, I didn’t know that xxxRuntimeConfig was deprecated

eps1lon commented 1 year ago

I believe the appDir already works as many in here want it to be: process.env.* is not statically baked into the final bundle at build time but read at runtime:

  1. a simple text search in .next did not show the secrets
  2. next build with secrets and next start without secrets caused missing permissions error. But then restarting next start with secrets caused no permission errors. Which makes me believe the secrets are not injected at build time.

Beta docs are missing the page for environment variables at the moment. Though the docs for edge runtime make no mention of injecting envrionment variables at runtime; only for build:

You can use process.env to access Environment Variables for both next dev and next build.

volnei commented 1 year ago

I'm facing the same problem.. I think the use of build time variables would be optional.

marekzan commented 1 year ago

I had the same issue when using env vars in different stages (staging, production) which were set via secrets in a kubernetes pod.

nextjs evaluates process.env calls at build time even in api routes in my case. If for example I have a dynamic url I want to use for a logout call "${process.env.NEXT_PUBLIC_IAM_ISSUER}/logout..." and I build the project, I can see that the relevant part in the built api file logout.ts now looks like "https://my-iam-issuer/logout..." So it won't be evaluated at runtime depending on the stage it runs in.

I wrote a small function to evaluate the env var during runtime:

const env = (key: string) => {
  const value = process.env[key];
  if (!value) {
    throw new Error(`requiered env var ${key}`);
  }
  return value;
};

This way the build output now shows "${env("NEXT_PUBLIC_IAM_ISSUER")}/logout..." And the correct url is being evaluated at runtime.

Talent30 commented 1 year ago

I had the same issue when using env vars in different stages (staging, production) which were set via secrets in a kubernetes pod.

nextjs evaluates process.env calls at build time even in api routes in my case. If for example I have a dynamic url I want to use for a logout call "${process.env.NEXT_PUBLIC_IAM_ISSUER}/logout..." and I build the project, I can see that the relevant part in the built api file logout.ts now looks like "https://my-iam-issuer/logout..." So it won't be evaluated at runtime depending on the stage it runs in.

I wrote a small function to evaluate the env var during runtime:

const env = (key: string) => {
  const value = process.env[key];
  if (!value) {
    throw new Error(`requiered env var ${key}`);
  }
  return value;
};

This way the build output now shows "${env("NEXT_PUBLIC_IAM_ISSUER")}/logout..." And the correct url is being evaluated at runtime.

It is bit confusing but I have to say it is working as expected, if you don't want the env vars to be baked into bundle you need to remove NEXT_PUBLIC_ because it is talking the bundler that this env var needs to be accessed on the client side(browser) where there is no process.env

marekzan commented 1 year ago

Thank you very much, this worked. I should've read the docs slower...

mltsy commented 1 year ago

This is indeed confusing! The note in question was added in this PR: https://github.com/vercel/next.js/pull/20869, addressing this issue: https://github.com/vercel/next.js/issues/19420.

I think perhaps it only applies to environment variables that begin with NEXT_PUBLIC_ and should be moved to the NEXT_PUBLIC_ section of the page (which is, itself, confusing to me, but that's another issue entirely).

@lachlanjc - any chance you could confirm this suspicion (as the original author of the PR)? Based on the comments above, it seems like @richardscarrott is correct about most environment variables being evaluated at runtime and working just fine with destructuring.

mltsy commented 1 year ago

(@timneutkens maybe you could also provide input/confirmation here, having approved that PR?)

manovotny commented 1 year ago

The Next.js have been completely rewritten since this issue was created. The documentation now states:

CleanShot 2023-06-14 at 16 19 08@2x

Which I believe clarifies the original ask of this issue.

Certainly happy to make adjustments if any of you read it and still have suggestions for things that are still unclear or confusing. Please leave a comment or open a PR (since the docs are now open source), but I am going to mark this issue as closed for now.

eps1lon commented 1 year ago

@manovotny getStaticProps seems to imply secrets are read at build time without making any mention of runtime which we're asking for here. This issue is not resolved for me by these docs.

So it would be nice to clarify if it's indeed intended by Next.js that process.env.SOME_SECRET is not replaced at build time if SOME_SECRET does not exist. That's certainly behavior I'd need.

mltsy commented 1 year ago

@eps1lon - So you're just looking for a note confirming that, something like "any variables in the runtime environment that don't appear in a .env file will be passed through to process.env as usual, unmodified"?

nick4fake commented 1 year ago

@mltsy It is extremely confusing. Even after reading docs multiple times I still don't understand what env vars and when are baked or can be accessed in runtime. And what about client code? It seems that all of them are pre-baked, so documentation is very lacking

g-monroe commented 1 year ago

I want to keep the discussion alive as currently I'm tackling the concept "Build Once, Deploy Anywhere". I ran multiple different tests with NextJs 13.4.4. These tests involved creating:

A .env that looks like this: ..... NEXT_PUBLIC_FRONTEND_PATH=dev-test

A next.config.js that looks like this: require("dotenv").config(); .... const nextConfig = { env: { NEXT_PUBLIC_FRONTEND_PATH: process.env.NEXT_PUBLIC_ORION_FRONTEND_PATH, }, .... basePath:/${env('NEXT_PUBLIC_FRONTEND_PATH')}, output: "standalone", (I've tried both @marekzan, and another dev's work for testing)

Finally running next build, and testing the findings by replacing the .env & running next start, each time it bakes in the path making it impossible for the concept "Build Once, Deploy Anywhere" if the path of the next app is in a different dns path.

For other env vars that don't involve the next.config.js there are clear work arounds like @marekzan solution. Is there any plans support dynamic env variables without them being baked in?

mltsy commented 1 year ago

@nick4fake - I was not trying to be dismissive, just trying to find a documentation update that might alleviate @eps1lon's confusion. It sounds like you have other questions/confusion about env-related processes. I agree that it's a complex system to understand - and so kind of hard to document in a simple way.

If you can outline some specific questions you have that are not answered by the documentation, maybe I could help, or you could open a new issue specifically with those questions?

To try to clarify based on what you said above, generally speaking:

mltsy commented 1 year ago

@g-monroe I agree this is not really supported by the next framework going forward. We are currently still using the legacy publicRuntimeConfig for this purpose, which I hope continues to be supported, because as far as I can tell, the alternative is to roll your own env variable API and load them manually from the client. I haven't opened an issue for this yet, since publicRuntimeConfig still works for our purposes, but it's certainly something I agree should be supported by the framework and not something each team has to reinvent for themselves.

timneutkens commented 1 year ago

To clarify this:

publicRuntimeConfig / serverRuntimeConfig both were created before static rendering existed in Next.js and don't line up well because of that. In case you want all routes to dynamically render so that you don't have to provide the environment variable during build you can add export const dynamic = 'force-dynamic' in the root layout but keep in mind that opts you out of all server-side caching layers.

Alternatively you can create a wrapper function that handles opting out of static rendering only when the environment variable is read. I.e.:

import 'server-only' // Not needed per-se in this case, but it allows you to block this file from being imported in client components.
import { headers } from 'next/headers'
function getMyApiKey() {
   headers() // Calling this to opt-out of static rendering
   return process.env.MY_API_KEY // As said above this is not inlined into the bundle so there's no risk of accidentally leaking it that way.
}
HuiSF commented 12 months ago

Thanks for the clarification @timneutkens this is helpful to understand an issue I'm facing. I have a follow up question, however, e.g.

Reading process.env.MY_API_KEY in getServerSideProps -> MY_API_KEY Needs to be available at runtime

How to ensure MY_API_KEY is available at runtime please?

Here's the issue I'm facing: Following this documentation, with the Pages Router, I added a key-value pair, say testKey: 123 pair under the env in the next.config.js.

When I attempted to access the value via process.env.testKey inside getServerSideProps, it returned undefined. It also returned undefined when attempted to access within a Page component, while I was seeing process.env[__NEXT_PRIVATE_RENDER_WORKER_CONFIG] has my key-value pair. This only worked in the middleware as process.env.testKey is replaced with the actual value in the script bundle. It's also noticeable that process.env[__NEXT_PRIVATE_RENDER_WORKER_CONFIG] is not available at build time within getStaticProps.

Using the same setup in the App Router, however, process.env.testKey works everywhere as described in the documentation.

fellipefreiire commented 12 months ago

I fail to understand how this is still a problem after 1 year, the docs aren't clear enough, the env vars on runtime doesn't seems to work properly, i would really appreciate if Next team took time to build some production ready product instead of something that only works with next dev...

sgoodrow-zymergen commented 11 months ago

@timneutkens Thank you for the case-by-case clarification.

Can you also clarify where dynamic API routes fall into this picture?

I have a pretty simple use-case where I'd like to include the value of process.env.ENVIRONMENT set in the deployment environment in my healthcheck endpoint (eg "production", "development", "staging", etc).

I currently have a config that attempts to read this value from process.env, which is then used in my healthcheck endpoint, but the value for environment is always undefined.

// config.ts
import "server-only";

const unknown = "unknown";

export const serverConfig = {
    name: process.env.npm_package_name || unknown,
    version: process.env.npm_package_version || unknown,
    environment: process.env.ENVIRONMENT || unknown,
    // other config variables
    ...
}

// route.ts
export const GET = async () => {
  return NextResponse.json({
    name: serverConfig.name,
    version: serverConfig.version,
    environment: serverConfig.environment,
    status: "OK",
  });
};

GET /api/health:

{"name":"my-ui","version":"0.2.17","environment":"unknown","status":"OK"}

FWIW:

timneutkens commented 11 months ago

Based on the code shared that route handler is static, doesn't use request values and no export const dynamic = 'force-dynamic'.

sgoodrow-zymergen commented 11 months ago

Based on the code shared that route handler is static, doesn't use request values and no export const dynamic = 'force-dynamic'.

Thank you for the reply!

Ah of course. That makes sense. It wasn't intuitive to me that reading from an environment variable (a choice I generally make in order to make some part of my application dynamic) would be treated by auto as a static attribute and lead to the route being static.

mltsy commented 11 months ago

So, reviewing @timneutkens great clarifications, and going back to @g-monroe's issue/example... I'm wondering if it would work to remove the NEXT_PUBLIC_ from that NEXT_PUBLIC_FRONTEND_PATH variable (so that it doesn't get replaced by next at build time) and then use getServerSideProps in the "page" file(s) wherever you might need to access that variable in client code?? (I'm not totally clear on why it needed to be NEXT_PUBLIC_ to begin with, so I may be missing something about how/where it's being used that complicates things, but if I'm understanding correctly, it seems like getServerSideProps is the Next.js solution for passing env data to the client) 🤔

(Or perhaps, if you're using the App Router rather than the Pages Router, and if it's a thing that only needs to be loaded once, it could be loaded into a ContextProvider in the Root Layout file which is rendered as Server Component?)

Jackbennett commented 6 months ago

Is there something to triple check the component is getting rendered as a server component? This variable is undefined whereas I expect it to work from the docs;

import { type ComponentProps, useMemo } from "react"
import { unstable_noStore as noStore } from 'next/cache'
import { MessageBox } from '@/components/AppOnloadMessage'

// const msg = global.process?.env?.APP_ONLOAD_MESSAGE // this works too
const msg = process.env.APP_ONLOAD_MESSAGE
console.log(`server render message: ${msg}`) /// prints "some test variable" in server output

export const AppOnloadMessage = (props: Omit<ComponentProps<typeof MessageBox>, "message">) => {
    noStore()
    console.log(`component render inner message: ${msg}`) /// prints undefined in browser dev console
    return useMemo(() => (msg ? <MessageBox message={msg} {...props} /> : undefined), [msg])
}

I did try without useMemo too, same result from APP_ONLOAD_MESSAGE='some test variable' yarn run dev

docs https://nextjs.org/docs/pages/building-your-application/configuring/environment-variables#runtime-environment-variables:

To read runtime environment variables, we recommend using getServerSideProps or incrementally adopting the App Router. With the App Router, we can safely read environment variables on the server during dynamic rendering. This allows you to use a singular Docker image that can be promoted through multiple environments with different values.

import { unstable_noStore as noStore } from 'next/cache'

export default function Component() {
  noStore()
  // cookies(), headers(), and other dynamic functions
  // will also opt into dynamic rendering, making
  // this env variable is evaluated at runtime
  const value = process.env.MY_VALUE
  // ...
}

The docs lead me to believe that really should work, I don't have 'use client' in there. Server components can be under client components, I thought.

mltsy commented 6 months ago

@Jackbennett - Oh wow, this is great news to see how the App Router (purportedly) facilitates this (thanks for the docs link)! I don't know that I'm quite grokking how it all works yet, but have you tried moving the process.env.VARIABLE reference inside the component function? I'm not totally clear on where/when each part of this module gets evaluated, but I'm curious if it's just the component function itself that is "dynamically rendered" on the server, since that's where you call noStore() to trigger dynamic rendering (while the module's root scope might be evaluated in the client?)

hos commented 4 months ago

Using some other solution on the internet, I came up with this.

https://gist.github.com/hos/9ac10156c6b63f7374032b1b16500d50

ritingliudd01 commented 4 weeks ago

Created a repo based on the solution from @hos

Demo: https://github.com/ritingliudd01/nextjs-14-get-runtime-env