Public PAYLOAD_PUBLIC environment variables don't work in production builds

[hdodov]

Bug Report

I want to use a public environment variable for the S3 URL of my app. I have this in my .env file:

PAYLOAD_PUBLIC_S3_URL=https://my-bucket.s3.eu-central-1.amazonaws.com

…and this in my src/collections/Media.ts file:

adminThumbnail: ({ doc }) => {
    let media = doc as unknown as MediaType;
    return `${process.env.PAYLOAD_PUBLIC_S3_URL}${media.sizes.thumb.url}`;
},

Everything works great when I npm run dev. The served URLs in the panel are these:

https://my-bucket.s3.eu-central-1.amazonaws.com/media/my-image-350x233.jpg

But when I make a production build with npm run build and then open the app with npm run serve, I get these URLs:

undefined/media/my-image-350x233.jpg

When I search for my bucket URL in the resulting build folder, it's nowhere to be found.

Other Details

Payload version 1.3.0

[JarrodMFlesch]

Hey @hdodov, if you console log process.env in your server file, after building, do your env variables log to the console?

[hdodov]

Yes. I added a log right after the dotenv require in my src/server.ts:

require("dotenv").config();
console.log(process.env);

// ...

When I create a production build with npm run build, then start the server with npm run serve, I can see the PAYLOAD_PUBLIC_S3_URL variable logged in the console, but it's not available in the build folder.

Here's what's inside the resulting build/main.38d9279b570606719248.js file:

adminThumbnail:e=>{let{doc:t}=e,n=t;return`${c.env.PAYLOAD_PUBLIC_S3_URL}${n.sizes.thumb.url}`},

What I add console.log(c.env); directly before the return keyword above, I only see an empty object {} in the browser console.

[JarrodMFlesch]

Sure, how are your files structured? Like this?

.env
/src
/build

[hdodov]

Yep, .env is at the project root, the same way it is when a project is generated.

[milamer]

I had the same problem when trying to use PAYLOAD_PUBLIC_ env variables in my payload.config.

If the variable is needed on the client and not on the server it is only replaced if it is used in the payload.config.ts file.

[JarrodMFlesch]

@hdodov @milamer beautiful. I was able to recreate it, thank you both!

[JarrodMFlesch]

OK! So, you need to add the following at the top of your payload config file.

import path from 'path';
import dotenv from 'dotenv';

dotenv.config({
  path: path.resolve(__dirname, '../.env'),
});

If you were fetching your env variables from a remote source you would need to wire up a custom build script that uses payloads build script after fetching your remote variables, something like this:

// customAdminBuild.js
const path = require('path');

// Tell Payload where to find your Payload config - can do it either here, or in package.json script
process.env.PAYLOAD_CONFIG_PATH = path.resolve(__dirname, 'src/payload.config.ts');

const { build } = require('payload/dist/bin/build');

const buildPayloadAdmin = async () => {
// Fetch your environment variables here
// And then run `build`

build();
}

buildPayloadAdmin();

And you would use that in your build command instead of payload build 👍

[brachypelma]

@JarrodMFlesch , than you for posting your solution to this problem. Unfortunately, I cannot get this to work. This may just be a misunderstanding on my part. Please let me know if it is.

My goal is to have separate .env files, one on my local computer for local development and one on my server for production use. I want to be able to have different serverURL and CORS config options in my payload.config for local and production development, so I would like to read these payload.config values from the .env files so the same config file will work in each environment.

Here is an example app I put together following your guidelines above.

https://github.com/brachypelma/payload-env-test

My steps to create this:

1. I ran npx create-payload-app with TS and blank template options
2. I added the following line to my .env file: PAYLOAD_PUBLIC_SERVER_URL=http://localhost:3000
3. I added the following to the top of my payload.config file:

   
   import * as dotenv from 'dotenv'

dotenv.config({ path: path.resolve(__dirname, '../.env'), })

4. I changed the buildConfig as follows:

export default buildConfig({ serverURL: process.env.PAYLOAD_PUBLIC_SERVER_URL, ... });

When I save this and run `npm run dev` and then load `http://localhost:3000/admin` in my browser, I get a blank screen and this in the console:

Uncaught TypeError: dotenv__WEBPACK_IMPORTED_MODULE_2__.config is not a function
    ts payload.config.ts:7

I'm not sure why dotenv.config is not recognized as a function, especially since it is called in server.ts

FWIW, I tried removing the dotenv.config call from payload.config and added the specified option to the dotenv.config call in server.ts, i.e.:

require('dotenv').config({ path: path.resolve(__dirname, '../.env'), });

However, when I do that, .env variables loaded in payload.config are undefined when I build the app.

Can you offer any help? I'm not really sure what I am doing wrong here. Apologies again if I just misunderstood your solution.

[JarrodMFlesch]

@brachypelma can you try using import instead of using require statement? I do get the blank screen error when using require to import dotenv.

import path from 'path';
import dotenv from 'dotenv';

dotenv.config({
  path: path.resolve(__dirname, '../.env'),
});

With this I am able to run dev and build, and read env vars in both. Let me know!

[brachypelma]

Thank you, @JarrodMFlesch. I tried this out, and it is working now. In case others in the future get hung up on this, I should specify that in order to get this to work, I added the following to the top of server.ts:

import dotenv from 'dotenv';

dotenv.config();

and this to the top of payload.config.ts:

import path from 'path';
import dotenv from 'dotenv';

dotenv.config({
  path: path.resolve(__dirname, '../.env'),
});

[hdodov]

@JarrodMFlesch thanks a lot for the help, but I think this is more of a workaround, rather than a solution.

First of all, I use environment variables in my src/server.ts file. If I put dotenv there, rather than in src/payload.config.ts, PAYLOAD_PUBLIC variables still don't work. So they must be in the Payload config. But if I put them there, my server.ts stops working, because environment variables are not defined yet. This forces me to load dotenv in both files, which is hacky and confusing.

Also, the dotenv usage states that an .env file is expected at the root of the project. About the path option:

Specify a custom path if your file containing environment variables is located elsewhere.

…but my .env file is not located elsewhere. As a regular dotenv user, this makes no sense.

Additionally, they advise:

As early as possible in your application, import and configure dotenv

…but the earliest possible place to load dotenv is in server.ts, not payload.config.ts, and if I load it in server.ts - PAYLOAD_PUBLIC variables don't work, as I've already said.

I think this issue should be fixed at a framework level.

[hdodov]

Actually, the problem seems to be that I was loading dotenv in server.ts, rather than in payload.config.ts. If I don't use the path option, it still works as expected.

It seems that in my situation, the fix was to simply add this at the top of my payload.config.ts:

import dotenv from "dotenv";
dotenv.config();

Note, however, that doing it with require won't work:

require("dotenv").config();

It'll give the following error in the browser:

Uncaught TypeError: n(...).config is not a function

---

Still, though, having to load dotenv in both server.ts and payload.config.ts is unexpected and will be a pitfall for a lot of users, in my opinion.

[hdodov]

Also, the docs recommend this:

We suggest using the dotenv package to handle environment variables alongside of Payload. All that's necessary to do is to require the package as high up in your application as possible (for example, at the top of your server.js file), and ensure that it can find an .env file that you create.

…but putting it at the top of your server.js file would break PAYLOAD_PUBLIC variables.

[JarrodMFlesch]

@hdodov in what way are you using it in the server.js file that it breaks for you? I am using it in a few projects, declaring the dotenv import in both places and it has been working for me, maybe you are referring to something in specific?

[hdodov]

@JarrodMFlesch I'm doing this in server.ts:

require("dotenv").config();

…and this in payload.config.ts:

import dotenv from "dotenv";
dotenv.config();

…and it works alright. My main concern is this:

I am using it in a few projects, declaring the dotenv import in both places and it has been working for me

The user shouldn't have to include dotenv in both places. That was my point - the docs I quoted suggest loading dotenv in server.ts, but doing so (without also loading it in payload.config.ts) won't work.

It'd be great DX if PAYLOAD_PUBLIC variables can work without having to load dotenv twice.

[CallMeLaNN]

@JarrodMFlesch possible to add the above fix and example into the create-payload-app payload.config.ts?

Talking about DX, here's the long story of my DX:

1. I run yarn dlx create-payload-app as usual with the blog template.

2. Obviously I have to change the hard coded serverURL with process.env.SERVER_URL but it doesn't work until I realize payload.config.ts will be use by both node and browser environment indirectly from this doc Config overview and Using environment variables in your config:

   This file is included in the Payload admin bundle, so make sure you do not embed any sensitive information.

   you need to make sure that your Admin panel code can access it

   My bad, I didn't get that Payload admin or Admin panel is the frontend admin UI in the first place. It is clear after reading the Webpack doc about stripe example using node packages.

3. Then I follow Webpack - Admin environment vars to have PAYLOAD_PUBLIC_ prefix. So it is now working when running yarn dev. Example from Generating TypeScript Interfaces also shows the same thing, so I'm on track.

4. After some development. I try yarn build before deploy and it failed due to the serverURL: process.env.PAYLOAD_PUBLIC_SERVER_URL with this error:

   Module not found: Error: Can't resolve 'process/browser'

   I can't find related issue so I start digging into Payload webpack config.

5. I realize I can't customize admin.webpack config to add EnvironmentPlugin or DefinePlugin plugin because it is for node environment and I simply can't do it in payload.config.ts! I'm stuck, then I found this issue and the last comment from @hdodov is finally working for me.

[JarrodMFlesch]

@hdodov wow I never responded - my bad, I totally thought I did.

The reason it must be declared in both places is because:

• when you serve, your project needs access to env in the admin panel
• when you build, your server code needs access to the env

I am not sure there is a single solution to allow for both to be done without declaring it in both.

@CallMeLaNN Yes I think that the example should be updated. I think we should add the import statements for dotenv in both places. Would you like to contribute with a simple PR to the 3 src/templates/[template-name]/src/payload.config.ts files? (see them here)

examaple (blog template):

import { buildConfig } from 'payload/config';
import path from 'path';
import Categories from './collections/Categories';
import Posts from './collections/Posts';
import Tags from './collections/Tags';
import Users from './collections/Users';
import dotenv from 'dotenv';

dotenv.config();

export default buildConfig({
  serverURL: process.env.PAYLOAD_PUBLIC_SERVER_URL || 'http://localhost:3000',
  admin: {
    user: Users.slug,
  },
  collections: [
    Categories,
    Posts,
    Tags,
    Users,
  ],
  typescript: {
    outputFile: path.resolve(__dirname, 'payload-types.ts')
  },
  graphQL: {
    schemaOutputFile: path.resolve(__dirname, 'generated-schema.graphql'),
  },
});

[MarkAtOmniux]

I'm not sure if 1.6 messed with something, but having followed this thread the environment variables are no longer being read in properly at startup.

I am using Google Cloud Run and passing Environment variables to it. This was working fine when I included this in my server.ts;

require('dotenv').config({ path: '.env' });

I also removed the dotenv.config() call in my payload.config.ts file.

The only issue with this is that my circumstance is the exact same with @hdodov, I cannot access PAYLOAD_PUBLIC inside my collection file. I'm not really sure what to do at this point. I tried everything mentioned before but it still does not work in both scenarios

[JarrodMFlesch]

@MarkAtOmniux is your issue in development, or when serving the built code?

Did you follow the upgrade guide in the 1.6 changelog?

[MarkAtOmniux]

Hi Jarrod - The issue is when serving the built code. I did follow the upgrade guide, yes. I didn't need to migrate any data as this was technically a clean install and deploy to my staging server

[JarrodMFlesch]

@MarkAtOmniux and you are using importing dotenv in both the payload config and the server file as mentioned above?

[MarkAtOmniux]

I was, yeah. For some reason that stopped the payload.config.ts file from reading in any new values passed at startup, it would instead default itself to the values in the .env file.

When I didn't include the dotenv in the payload config file, it would start reading in values (PAYLOAD_PUBLIC) at startup inside the payload config, but not in any of my collections

[MarkAtOmniux]

I've done a bit of experimenting.

When using dotenv.config() in the server.ts file AND payload.config.ts file any env variables used in the server.ts file are completely static at startup. The variables will be read in from the .env file and cannot be overridden.

However, environment variables used in the payload.config.ts file CAN be overwritten at startup. Here is some of my code;

import { buildConfig } from 'payload/config';
import dotenv from 'dotenv'

dotenv.config();

export default buildConfig({
  serverURL: process.env.PAYLOAD_PUBLIC_PAYLOAD_URL, // this can be changed at startup by passing in a new value
  collections: [
    Users,
    Pages,
    Media,
    ReusableContent,
    ContactSubmissions
  ],
  ...
  plugins: [
    cloudStorage({
      collections: {
        media: {
          disablePayloadAccessControl: true,
          disableLocalStorage: true,
          prefix: process.env.PAYLOAD_PUBLIC_GCS_PREFIX, // this can be changed at startup by passing in a new value
          adapter: gcsAdapter({
            options: {
              apiEndpoint: process.env.PAYLOAD_PUBLIC_GCS_ENDPOINT, // this can be changed at startup by passing in a new value
              projectId: process.env.PAYLOAD_PUBLIC_GCS_PROJECT_ID, // this can be changed at startup by passing in a new value
            },
            bucket: process.env.PAYLOAD_PUBLIC_GCS_BUCKET, // this can be changed at startup by passing in a new value
          })
        }
      }
    }),

import express from 'express';
import payload from 'payload';
import dotenv from 'dotenv';

dotenv.config(); // doing require("dotenv").config() also does not work

const app = express();

// Redirect root to Admin panel
app.get('/', (_, res) => {
  res.redirect('/admin');
});

// Initialize Payload
payload.init({
  secret: process.env.PAYLOAD_SECRET,
  mongoURL: process.env.MONGODB_URI,  // this cannot be changed at startup. Whatever value is present inside the .env file when the app is created will be used. Even attempts to use PAYLOAD_PUBLIC_MONGODB_URI would result in static data
  express: app,
  onInit: () => {
    payload.logger.info(`Payload Admin URL: ${payload.getAdminURL()}`)
  },
})

app.listen(8000);

.env

PAYLOAD_PUBLIC_PAYLOAD_URL=http://localhost:8000
PAYLOAD_PUBLIC_WEBSITE_URL=http://localhost:3000
PAYLOAD_PUBLIC_GCS_ENDPOINT=https://storage.googleapis.com/
PAYLOAD_PUBLIC_GCS_PROJECT_ID=my-project-id
PAYLOAD_PUBLIC_GCS_BUCKET=my-bucket
PAYLOAD_PUBLIC_GCS_PREFIX=dev

PAYLOAD_SECRET=testing123
MONGODB_URI=mongodb://localhost:27017

[CallMeLaNN]

@MarkAtOmniux I'm not sure the meaning of changed at startup but probably the differences you refer related to Webpack parse and replace process.env.PAYLOAD_PUBLIC_PAYLOAD_URL etc from payload.config.ts and (supposedly) all other imported files too during dev and build. I haven't try env on collection but check the position of dotenv.config();, make it highest possible.

@JarrodMFlesch Let me add into the template but IIRC there is one thing to check/test on it. I'll get back later.

[MarkAtOmniux]

@CallMeLaNN What I mean by 'changed at startup' is when calling 'yarn serve', either locally or in a docker container, the values passed will be ignored and instead the values set in the .env file will be used.

[remy90]

Payload + NextJS Server-Rendered TypeScript Boilerplate doesn't work with pnpm, even when both imports are included as specified above. Only change being the package manager:

ERROR in ./src/collections/Pages/hooks/revalidatePage.ts 153:117-124
Module not found: Error: Can't resolve 'process/browser' in '/src/collections/Pages/hooks'
 @ ./src/collections/Pages/index.tsx 5:0-70 16:94-106 32:12-26
 @ ./src/payload.config.ts 3:0-44 13:8-13
 @ ./node_modules/.pnpm/payload@1.6.24_typescript@4.9.5/node_modules/payload/dist/admin/Root.js
 @ ./node_modules/.pnpm/payload@1.6.24_typescript@4.9.5/node_modules/payload/dist/admin/index.js 10:31-48

ERROR in ./src/collections/Users.ts 8:20-27
Module not found: Error: Can't resolve 'process/browser' in '/src/collections'
 @ ./src/payload.config.ts 4:0-40 12:8-13
 @ ./node_modules/.pnpm/payload@1.6.24_typescript@4.9.5/node_modules/payload/dist/admin/Root.js
 @ ./node_modules/.pnpm/payload@1.6.24_typescript@4.9.5/node_modules/payload/dist/admin/index.js 10:31-48

Whats more confusing, the console.log in the snippet below from revalidatePage successfully logs PAYLOAD_PUBLIC_SERVER_URL:

import type { AfterChangeHook } from 'payload/dist/collections/config/types'
console.log(`process.env.PAYLOAD_PUBLIC_SERVER_URL: ${process.env.PAYLOAD_PUBLIC_SERVER_URL}`)
// ensure that the home page is revalidated at '/' instead of '/home'
export const formatAppURL = ({ doc }): string => { ... }

Switching back to yarn fixes the issue. @JarrodMFlesch Is it worth reopening this or starting a new issue?

[JarrodMFlesch]

@remy90 you should make a new issue with a reproduction repository (does not have to be in the _community folder as the issue template will ask you to do) so we can recreate this on our end.

[github-actions[bot]]

This issue has been automatically locked. Please open a new issue if this issue persists with any additional detail.