feat: better way to validate / import env vars

[LucasAlda]

Hello! I am moving a create-t3-app project to a turborepo based on this template, and I notice that the env checker remains in the next folder, contrary to what I would have thought since now it wouldn't be fully checking the env variables for other apps or packages. It would be great to add a new package that checks all as a dependency in the TurboRepo workflows, similar to the env folder in @acme/next.

[juliusmarminge]

Yes, if you have a good way?

[LucasAlda]

I was thinking on having the package really similar to the actual env folder, exporting the type checked env and adding a "envcheck" script that runs the zod.parse at the beginning.

I just did a quick research and I couldn't find a way to run a script before the turbo dev since that pipeline is run in parallel and doesn't wait for dependsOn. I can imagine right now to posible ways:

1. Instead of having "dev" (and build) depending on typecheck, renaming typecheck to dev and run it in parallel, but it won't be the first one and I not sure if it's going to stop the whole process having a non-zero exit on one package.
2. having the script dev: 'turbo typecheck && turbo dev' and build: 'turbo typecheck && turbo build'

I personally prefer the 2nd one but I'd love to hear your opinion. If we find a good way I'm totally disposed to code it and create the pr

[juliusmarminge]

I don't want typechecking to be a prerequisite to dev - you should be able to dev even when your app has typeerrors.

I am also having second thoughts about a global env package - what if you want to run your nextjs app but dont have all the credentials for Expo? a global package here would prevent you from running any app when in reality it's only the expo app which isn't "runnable"

I added a thought in #100 about it being each app's responsibility to check their env vars. this isn't an optimal solution since you can't import the validated (and maybe transformed) environment variables in for example the auth package.

Curious what the best approach here is.

[LucasAlda]

I don't want typechecking to be a prerequisite to dev - you should be able to dev even when your app has typeerrors.

I don't know why I write so many times typecheck, I was reffering to envchecking.

I am also having second thoughts about a global env package - what if you want to run your nextjs app but dont have all the credentials for Expo? a global package here would prevent you from running any app when in reality it's only the expo app which isn't "runnable"

That's a really interesting point. It would be fantastic to have that global env package so that all of the validation process for your monorepo is still unified on one package, but separated into "env modules" that are checked or not depending on which apps or packages are starting.

I'm thinking of a function that receives an array of the env keys with the validator and a dependency array that specifies the apps/packages that use it. Maybe it's too complicated, but it would have an amazing DX. Also, I can't think of how to get the apps/packages that are going to be run.

On the other hand, the package can abandon the concept of global check while still providing the configuration to check every app for the desired environments and get them typed safely.

[adambarito]

I tried pulling the env checking into a separate monorepo package this weekend. The package ('@acme/env') defines the destructured processEnv, and exports zod schemas for all client and server env variables. It also provides the checking function. Then any other app or package can define its own env.ts file that pulls in only what it needs (by picking variables from the client and/or server zod schema). That way, the env variables and zod schema are all kept in one place.

If this seems like the right direction I'd be happy to work towards a PR.

The '@acme/env' package:

// packages/env/index.ts (alias '@acme/env')
import { z, ZodRawShape } from 'zod';

const processEnv = {
    // CLIENT
    NEXT_PUBLIC_APP_BASE_URL: process.env.NEXT_PUBLIC_APP_BASE_URL,
    NEXT_PUBLIC_APP_DEV_PORT: process.env.NEXT_PUBLIC_APP_DEV_PORT,
    NEXT_PUBLIC_MAGIC_PUBLISHABLE_KEY: process.env.NEXT_PUBLIC_MAGIC_PUBLISHABLE_KEY,

    // SERVER
    NODE_ENV: process.env.NODE_ENV,
    DATABASE_URL: process.env.DATABASE_URL,
    NEXTAUTH_URL: process.env.NEXTAUTH_URL,
    NEXTAUTH_SECRET: process.env.NEXTAUTH_SECRET,
    DISCORD_CLIENT_ID: process.env.DISCORD_CLIENT_ID,
    DISCORD_CLIENT_SECRET: process.env.DISCORD_CLIENT_SECRET,
    MAGIC_SECRET_KEY: process.env.MAGIC_SECRET_KEY,
    POSTMARK_SERVER_API_TOKEN: process.env.POSTMARK_SERVER_API_TOKEN,
};

export const clientEnvAllSchema = z.object({
    NEXT_PUBLIC_APP_BASE_URL: z.string(),
    NEXT_PUBLIC_APP_DEV_PORT:
        process.env.NODE_ENV === 'development'
            ? z.string().min(1)
            : z.string().min(1).optional(),
    NEXT_PUBLIC_WEB_DEV_PORT:
        process.env.NODE_ENV === 'development'
            ? z.string().min(1)
            : z.string().min(1).optional(), 
    NEXT_PUBLIC_MAGIC_PUBLISHABLE_KEY: z.string(),
});

export const serverEnvAllSchema = z.object({
    NODE_ENV: z.enum(['development', 'test', 'production']),
    // INTERNAL
    API_USER_ID: z.string(),
    NEXTAUTH_URL: z.preprocess(
        // This makes Vercel deployments not fail if you don't set NEXTAUTH_URL
        // Since NextAuth.js automatically uses the VERCEL_URL if present.
        str => process.env.VERCEL_URL ?? str,
        // VERCEL_URL doesn't include `https` so it cant be validated as a URL
        process.env.VERCEL ? z.string() : z.string().url(),
    ),
    NEXTAUTH_SECRET:
        process.env.NODE_ENV === 'production'
            ? z.string().min(1)
            : z.string().min(1).optional(),

    // EXTERNAL
    DATABASE_URL: z.string().url(),
    DISCORD_CLIENT_ID: z.string(),
    DISCORD_CLIENT_SECRET: z.string(),
    MAGIC_SECRET_KEY: z.string(),
    POSTMARK_SERVER_API_TOKEN: z.string(),
});

export function zEnv<TClient extends ZodRawShape, TServer extends ZodRawShape>({
    clientEnvSchema,
    serverEnvSchema,
}: {
    clientEnvSchema: z.ZodObject<TClient>;
    serverEnvSchema: z.ZodObject<TServer>;
}) {
    const isServer = typeof window === 'undefined';

    const merged = serverEnvSchema.merge(clientEnvSchema);
    const parsed = isServer
        ? merged.safeParse(processEnv)
        : clientEnvSchema.safeParse(processEnv);

    if (!parsed?.success) {
        console.error(
            '❌ Invalid environment variables:\n',
            ...formatErrors(parsed.error.format()),
        );
        throw new Error('Invalid environment variables');
    }

    const env = new Proxy(parsed.data, {
        get(target, prop) {
            if (typeof prop !== 'string') return undefined;
            if (!isServer && !prop.startsWith('NEXT_PUBLIC_'))
                throw new Error(
                    `❌ Attempted to access server-side environment variable '${prop}' on the client`,
                );

            if (prop in target) return target[prop as keyof typeof target];

            return undefined;
        },
    }) as z.infer<typeof merged>; // not ideal to cast it this way, but should be typesafe due to isServer guard above

    return env;
}

const formatErrors = (errors: z.ZodFormattedError<Map<string, string>, string>) =>
    Object.entries(errors)
        .map(([name, value]) => {
            if (value && '_errors' in value) return `${name}: ${value._errors.join(', ')}\n`;
        })
        .filter(Boolean);

and an example of the api pulling in what it needs:

// packages/api/src/env.ts
import { clientEnvAllSchema, serverEnvAllSchema, zEnv } from '@barely/env';

const serverEnvSchema = serverEnvAllSchema.pick({
    NODE_ENV: true,
    OPENAI_ORG_ID: true,
    OPENAI_API_KEY: true,
});

const clientEnvSchema = clientEnvAllSchema.pick({});

const env = zEnv({ serverEnvSchema, clientEnvSchema });

export default env;

[LucasAlda]

I really like this method since it allows you to have the best of both worlds: a single env file and validation on @acme/env while maintaining the granularity of only checking the variables required for the running apps. All that is required for everything to be unified and typesafe is for each individual package or app to call zEnv at the beginning.

[adambarito]

@juliusmarminge any thoughts on this direction?

[juliusmarminge]

@juliusmarminge any thoughts on this direction?

Looked at it briefly but will have to take a closer look this weekend. Generally think it might work?

[adambarito]

It's cleaned up how I'm handling env variables in a few different monorepo apps + packages. Another iteration I'm liking is integrating the 'baseUrl' logic into the processEnv file (instead of in a shared util that I needed to import all over the repo).

const processEnv = {
    // CLIENT
    NEXT_PUBLIC_APP_BASE_URL:
        typeof window !== 'undefined'
            ? '' // browser should use relative url
            : process.env.NODE_ENV === 'production'
            ? `https://${process.env.VERCEL_URL}` // SSR should use vercel url
            : `http://localhost:${process.env.NEXT_PUBLIC_APP_DEV_PORT}`, // dev SSR should use localhost

    NEXT_PUBLIC_WEB_BASE_URL:
        typeof window !== 'undefined'
            ? ''
            : process.env.NODE_ENV === 'production'
            ? `https://${process.env.VERCEL_URL}`
            : `http://localhost:${process.env.NEXT_PUBLIC_WEB_DEV_PORT}`,

    NEXT_PUBLIC_LINK_BASE_URL:
        typeof window !== 'undefined'
            ? ''
            : process.env.NODE_ENV === 'production'
            ? `https://${process.env.VERCEL_URL}`
            : `http://localhost:${process.env.NEXT_PUBLIC_LINK_DEV_PORT}`,

    NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: process.env.NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY,

    // SERVER
    NODE_ENV: process.env.NODE_ENV

        /// ...rest of env variables

Then I'm not conditionally pulling in ports and/or production urls anywhere, and the clientEnvSchema just looks like this:

export const clientEnvAllSchema = z.object({
    NEXT_PUBLIC_APP_BASE_URL: z.string(), // for apps/app
    NEXT_PUBLIC_WEB_BASE_URL: z.string(), // for apps/web
    NEXT_PUBLIC_LINK_BASE_URL: z.string(), // for apps/link
    NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: z.string().min(1),
});

[juliusmarminge]

Playing around with a bit of an alternative approach that will let you do this in each app/pkg:

Should have something up by tomorrow

[juliusmarminge]

Checkout #135 and leave suggestions

[harry-peirse]

Hey - Thought I'd add my 2c. I really like this idea - but problem I'm having when I try to add it to my project is that I'm forced to import it as an ESM module, whereas my other packages projects don't give me the same grief. In my instance I'm wanting to create a regular Node Express app along side the Next app, and use the env variables in both. My knowledge of working around the CJS/ESM is certainly limited (and it's a complex area) so perhaps I'm missing something - but if you're aware of a way to keep this package not tied to being an ESM to my other packages that would be ideal. Thanks!

[LucasAlda]

Congratulations, @juliusmarminge, on the release of t3-env!! It's incredible how you can turn a simple issue I raised 5 months ago into a great, agnostic, and simple package. You're truly on another level.

[Tobjoern]

Congratulations, @juliusmarminge, on the release of t3-env!! It's incredible how you can turn a simple issue I raised 5 months ago into a great, agnostic, and simple package. You're truly on another level.

t3-env looks very promising! Do you know if it'll be integrated into the t3-turbo repo?