search

gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.21k stars 10.33k forks source link

Single build for multiple environment #22634

Closed Tush-r closed 4 years ago

Tush-r commented 4 years ago

Summary

Hello. For our gatsby project, we'd like to build it once and use it for multiple environments( say testing and staging). Only the environment variables are different for each environment.

Say our .env file looks like this,

SITE_URL=https://testorstage.willbedifferentforeach.env
SOME_API_KEY=abc1212xyz
MY_API_ENDPOINT=https://api.mysite.com
SOME_OTHER_KEY=qwertyabcd

Possible approaches

  1. Loading the env vars from a static/env.js script:
    • We create a static/env.js and declare the env vars in it( which will load them in globalThis of the browser).
    • Then we use the wrapRootElement of browser api to wrap the app in React.Context and load the script from there.
    • Works well for env vars that are required in the react components.
    • Does not work for SITE_URL var which is used by robot-txt plugin.
  2. Re-running webpack on the public folder:
    • We replace the environment variables by some unique placeholders, like, 
      SITE_URL=https://some.placeholder.url
SOME_API_KEY=SOME_API_KEY_PLACEHOLDER
    • Re-running webpack with DefinePlungin and some HTML/ text replacement plugin, replacing the placeholder string with the real keys(or text).
    • I have not tried this, as I am not sure how the wepack config would look like.
    • webpack-stats.json does look like an interesting file to get the webpack entry. Any help on this front would be appreciated.
  3. Running CLI tools on the public folder:
    • Similar to the above approach.
    • We replace the environment variables by some unique placeholders, like, 
      SITE_URL=https://some.placeholder.url
SOME_API_KEY=SOME_API_KEY_PLACEHOLDER
    • Using some cli tools( say sed & grep or others), replace placeholder strings with actual env vars.
    • Currently testing, if this works.

Questions

  1. Is there a better way to do this?
  2. Will replacing strings in generated js/ html files cause any issue?
vladar commented 4 years ago

Have you tried gatsby-env-variables plugin? Even if it doesn't work for you, you can still use it as a reference for webpack's DefinePlugin usage with Gatsby.

geemanjs commented 4 years ago

Thanks @vladar..

We have a similar requirement to the above and were looking for the "correct" way to go about solving it.. Everything we've come across so far requires us do something at the point gatsby build is ran.

Of our setup the important part is we use the gatsby contentful plugin to pull in blog posts at build time.

We have a dev, stg, prd environment and we use "GitOps" for our deployments. Currently our build pipeline is:

  1. Build the application & package the public directory into an nginx docker image
  2. Deploy the docker image to dev
  3. Deploy the docker image to stg
  4. Deploy the docker image to prd

This works for us and everytime a new blog post is published we retrigger back at step 1 so we can see how it looks in dev/stg before promoting to prd.

We've recently added a new analytics tool which gives us a "test" and "live" token. The "Test" for our dev, stg environments and the "live" for our prd ones. This is where the requirement for us having different environment variables on the client side depending upon the runtime environment comes from. (these tokens are meant to be public)

I can understand the difficulties of this when the output of gatsby build is a collection of static files but am unsure if there is a pre-existing way within the gatsby ecosystem to achieve our goal.

We initially went down setting the environment variables at the OS and then going down option 2 above (by running yarn build at application startup before moving into nginx) but this would also result in new contentful content being pulled into the site which we don't want. Plus we want to be able to rollback to a previous version at any time.

We almost need a different "post processing" step that would allow us to substitute environment variables into the JS after gatsby build has been executed however I'm guessing this isn't really an option because of webpack minification.

Webpack minifiction also rules out option 3 above. Option 3 is good for strings but when booleans are in use code may be removed depending upon the value.

Right now the only way I can see this fitting into our pipeline is to use gatsby-env-variables and run gatsby build with a different environment param and copying the public directory into an <environment> directory then passing an environment variable into nginx to use a different image.

GATSBY_ACTIVE_ENV=dev && gastby build
cp -R public/ /opt/app/dev

GATSBY_ACTIVE_ENV=stg && gastby build
cp -R public/ /opt/app/stg

GATSBY_ACTIVE_ENV=prd && gastby build
cp -R public/ /opt/app/prd

Then on startup using a passed in environment variable ACTIVE_ENV to trigger moving the directory to the one that is "served" by nginx.

cp /opt/app/${ACTIVE_ENV} /usr/share/nginx/html/

Hope you can suggest a better way.. James

Tush-r commented 4 years ago

@vladar Sorry for the late reply. @Geeman201 did a better job of explaining the problem that we both have.

Currently, I'm looking at the possibilities of approach 3, to make it work for all kinds of env vars.

github-actions[bot] commented 4 years ago

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open! As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

github-actions[bot] commented 4 years ago

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

Miyou commented 3 years ago

Can we please reopen this and get some response from the Gatsby team? I'm having the same issue and I feel like someone in the Gatsby team must have encountered this and have a good solution for it.

bengro commented 3 years ago

This is not a Gatsby issue, but a common challenge of any web application.

Having multiple images for different environments doesn't scale well and is an anti-pattern.

To avoid it, the problem is commonly solved in two popular ways:

1) Fetch configuration from an endpoint. The web server that deploys the app also provides a config endpoint. Because both app and config endpoint are running under the same host, the app can hardcode it. E.g. /v1/config. 2) Inject config into static files at runtime by server software. Web servers provide means to replace strings and have access to environment variables. See, SSR/nginx example.

More concretely, for Gatsby this worked very well for me:

1) Base the docker image on nginx and provide custom nginx.conf to turn on SSR. 2) Build Gatsby as normal and copy production build to image. 5) Provide an entry point for the container, which copies the environment variable to a file in the public folder, e.g. echo ${VAR} > /usr/share/nginx/html/var 3) In the app, provide an config abstraction which picks up configuration. When process.env.VAR is not available, fall back to window.VAR. 4) Define .env.development and define your environment variables for development mode. 6) Overwrite html.js to contain a script block defining the variables: e.g. VAR: '<!--#include file="config_var" -->',

MattFanto commented 3 years ago

I'm facing this problem as well and as Miyou I think it should be reopened. This is, of course, a common challenge of any web application, but some framework provides a solution out of the box (e.g. https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration).

It would definitely be helpful if an official solution is provided instead of relying on some custom solution

viralfineos commented 1 year ago

Is there any solution related to this topic?