Request: Support for Scoping by Branch

[Undistraction]

Is your feature request related to a problem? Please describe.

I think it would be useful to support environment-specific branches.

It is perfectly easy to set up Netlify to deploy multiple branches of the same project, so you can easily have three separate environments easily enough. In the following discussion I will use Gatsby, but I think this problem applies to whatever static site builder you are using.

• dev When running locally on localhost
• staging A staging branch added to Netlify as an additional branch deploy.
• production A production branch added to Netlify as the production branch.

If we weren't using Netlify CMS, this would be all we would need. We could develop locally and push to staging when we wanted to make changes available, then promote by merging staging into production and pushing to the production brancg when these changes are ready.

However, out of the box all these environments will be using the same config.yml, meaning all three will load from and make changes to the same branch - whichever branch you have defined as the value for branch. Assuming your branch is set to production then a change made by a dev locally is propagated to production which is obviously not tennable.

What is needed is a way to isolate content changes to each environment, so that changes made in development are made to a dev branch, changes made on staging are made to a staging branch and changes made on production are made to a production branch.

Describe the solution you'd like

I think it would make sense for Netlify-CMS to support the ability to enable automatic population of branch field with name of current Git branch, for example:

branch: auto

So assuming the staging branch is being deployed by Netlify to a staging URL, on staging branch is set to staging, meaning any changes made on staging are kept within the staging branch. It would probably make sense to allow more control over mappings of branch to environment, for example supporting some kind of map that decides the branch based on a regex run on the branch name, for example any branch prefixed with hotfix- would use the staging branch.

[Edit] Netlify offers the concept of 'Contexts' (here) which sound perfect for this. They can be used to set an Env based on branch. So all that is necessary is for Netlify CMS to check for a predetermined env, for example NETLIFY_BRANCH before falling back to the branch defined in the config.yml.

Describe alternatives you've considered

This can be currently only be achieved by deploying an environment-specific config.yml to each remote environment, and defaulting to a dev branch in development. Probably the easiest way to do this is to use a pre-build step to populate the branch field your config.yml with the name of the current branch, however this adds additional complexity and obfuscation. I think this should be declarable in Netlify's config.yml.

[talves]

I think this can be handled with some setup similar to what I did here. It would require more advanced knowledge of creating a custom setup. There are some other ways we may be able to setup using init.

[Undistraction]

@talves Looks like the branch is pulled from the config here in the Git-gateway backend and this pattern is followed in all backend implementations. Would be easy enough to place this access behind a functions which checks for an ENV first.

[talves]

I may be missing what you are asking for here.

I don't think the cms should determine the branch automatically. What I am saying is the branch can be set based on context or you could write a backend (custom) that does what you are asking for above to allow for a special settings use case.

[Undistraction]

I'm really just looking for a way to ensure that changes made in an environment are only applied to that environment rather than an arbitrary single branch shared by all environments. Netlify's Contexts allow you to say 'When you publish to this branch, set an ENV to something', so I was imaging that the something could be the name of the branch to pull and publish changes form for that environment, so:

1. I push my staging branch to my repo.
2. Netlify builds and deploys the site to staging--example.com.
3. Netlify sets NETLIFY_BACKEND_BRANCH env to staging via its context API.
4. When I launch Netlify CMS from staging--example.com/admin/ it looks for that env and sets the backend branch to its contents (staging).

Now I can edit content in the staging environment without effecting any other branch. And I can do the same for production.

[talves]

Then I did understand it right. I would just override the config.yml with a pre-process script during the build process. If I am integrating the cms into a react app, I would just use the code to set it within a manual init like the example I mentioned in my previous comment. Both of those solutions would use the context to determine the branch that is being built/deployed.

[Undistraction]

I would just override the config.yml with a pre-process script during the build process

Out of interest how would you approach this?

[talves]

Let's take this to https://gitter.im/netlify/NetlifyCMS for that discussion then we can edit the solution here.

[erquhart]

@talves closing this, but feel free to re-open if it merits further discussion here.

[Undistraction]

@erquhart I think this is still a valid request. The workaround is pretty convoluted and the behaviour is far from obvious to new users.

[talves]

@erquhart this issue is a matter for discussion prior to deciding whether we are going to support it within the bundle internal or just show how this can be done by extending the cms.

I don't see this as convoluted, because I believe the developer of the website should be the one deciding the config.yml change and not the CMS itself and it is pretty much supported using manual init at this time.

// Setting the branch based on an environment variable or some global
const branch = window.CMS_BRANCH || 'master'

// This global flag enables manual initialization.
window.CMS_MANUAL_INIT = true

// Usage with import from npm package
import CMS, { init } from 'netlify-cms'

// Usage with script tag
const { CMS, initCMS: init } = window

/**
 * Optionally pass in a config object. This object will be merged into
 * `config.yml` if it exists, and any portion that conflicts with
 * `config.yml` will be overwritten. Arrays will be replaced during merge,
 * not concatenated.
 *
 * For example, the code below contains an incomplete config, but using it,
 * your `config.yml` can be missing its backend property, allowing you
 * to set this property at runtime.
 */

init({
  config: {
    backend: {
      name: 'github',
      branch: `${branch}`
    },
  },
})

[erquhart]

Sounds like we still need to discuss, reopening.

[robsonsobral]

Duplicate?

[mrfoster]

I have been using the technique described here: https://www.netlify.com/docs/netlify-toml-reference/#caveats

e.g. edit netlify.toml to something like this

[context.production.environment]
  CMS_BRANCH = "production"

[context.staging.environment]
  CMS_BRANCH = "staging"

[build]
  base    = "frontend"
  publish = "frontend/public"
  command = "sed -i \"s/branch: production/branch: ${CMS_BRANCH}/g\" static/admin/config.yml && gatsby build"

[erquhart]

@robsonsobral https://github.com/netlify/netlify-cms/issues/1795#issuecomment-433070447

[erquhart]

@Undistraction this is a tough nut to crack. The big questions to me are:

• How do we make this work for any environment (not just Netlify)?
• How does it impact the editorial workflow (which creates branches and submits pull requests)?

Parsing an environment name from a url is a fair enough approach, and platforms like Netlify can handle this without configuration (eg. when using Git Gateway backend). But that still means you have a separate config on each branch, which means you can't merge staging straight to production. Are you only looking to automate the branch name?

[ghost]

Sorry to jump into this conversation late, but really liking the discussion above. I think what @Undistraction is proposing is awesome, and I want to see if I can help pick up the discussion again to come to an idea of how we'd want it implemented. Thanks @erquhart for posing these questions. I'll make a stab at answering at least the second one.

How does it impact the editorial workflow (which creates branches and submits pull requests)?

I would say we drop the config.backend.branch value all together. It doesn't seem necessary any longer. For backwards compatibility, we could also support the branch: auto format that @Undistraction originally proposed, which would signify that the user intends the branch to be determined by env var.

Either way, the branch should be determined by the environment variable that is set. To set that, I would simply set an environment variable of say CMS_BRANCH to whichever branch I intend content changes to be committed to. If this isn't set manually, it could default to master the way it does today.

In order to support the editorial workflow, I think would could simply create branches of the branch defined in the environment variable. To recap, today when content changes are made, a new branch is created called cms/<object_slug> and a PR is opened from cms/<object_slug> into master (or whatever branch is defined in the config.yml).

So, if CMS_BRANCH is currently set to staging and I make a content change to staging, then we would cut a new branch off of the current branch called cms/<parent_branch>/<object_slug>, which would be cms/staging/<object_slug> in this case. Then a new PR would be opened from cms/staging/<object_slug> into staging. When the draft content change is published, that PR is merged and the content is then live in the staging environment.

To promote content from staging to master (production), we could eventually create a UI to allow that, but for now it would be as simple as opening a PR from staging to master and merging it.

[erquhart]

Thanks for digging in @zboman!

I think what you're describing is already possible - you can dynamically define config.backend.branch by configuring the CMS with JavaScript, and the editorial workflow already only loads pull requests whose base branch matches config.backend.branch, without needing the environment hardcoded into the PR branch name.

[ghost]

@erquhart You are correct. I was able to move to manual initialization and accomplish essentially the same thing. It's a bit wonky but overall accomplishes the same thing.

[kara-todd]

@erquhart Sorry to necro this... but... :)

I have a situation where I need to change the base branch for the CMS and PRs opened based on the deploy preview. Here's my use-case. We setup new CMS functionality, pages, fields, etc. and then put them up on a deploy URL to QA.

Right now I can see where having a manual "QA" url would let me merge things in and test this with a manually defined base branch as explained above. However, what I'd really like is to use the current branch for the PR. So if I opened a pr for about-page (PR 12) then any CMS edits made on the url https://deploy-preview-12--project-name.netlify.com/ would use the about-page branch...

Does that make sense? Or is the already possible somehow I'm not seeing? I tried to find if Netlify exposed the branch somewhere, but I couldn't find anything concrete. :/

[erquhart]

@kara-todd for your use case and every other listed here, the recipe is:

1. Grab the branch from your build environment (Netlify provides a branch var among others)
2. Stash it somewhere, probably as a global variable on the window
3. Use that variable to set the branch in your config (you can extend your config.yml in js using manual init

Let me know if that makes sense.

[kara-todd]

@erquhart Thanks for the reply. I think where I am having some confusion on is where exactly to grab and set those variables... So for example right now I am using a process like this:

# netlify.toml

[context.qa.environment]
  GATSBY_CMS_BRANCH = "qa"

// cms.js (manual init)

import { init } from "netlify-cms-app";

// This global flag enables manual initialization.
window.CMS_MANUAL_INIT = true;

const { GATSBY_CMS_BRANCH } = process.env;

  const config = {
    backend: {
      name: "github",
      repo: "repo/name",
      branch: GATSBY_CMS_BRANCH || "master"
    }
  };

  init({ config });

This seems to be working... but what I would ultimately like is to always grab the branch variable you mentioned for every context.deploy-preview and context.branch-deploy. It wasn't clear to me how to access the branch variable in netlify.toml file. Is this syntax described somewhere?

Edit several months later...

To any future poor souls that end up here. My issue above is you cannot destructure environment variables

From the documentation:

Note: since Gatsby uses the Webpack DefinePlugin to make the environment variables available at runtime, they cannot be destructured from process.env; instead, they have to be fully referenced. GATSBY_API_URL will be available to your site (Client-side and server-side) as process.env.GATSBY_API_URL.

I also had to whitelist the default netlify variables which I was able to do using gatsby-plugin-env-variables.

// gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-env-variables`,
      options: {
        whitelist: ["BRANCH", "CONTEXT", "HEAD"]
      }
    }
 ]
};

[erquhart]

netlify.toml isn't involved - if you check the docs I linked, there's a list of environment variables that Netlify sets automatically for every build. One of them is BRANCH, so you can grab the current branch via process.env.BRANCH.

_{Sent with GitHawk}

[kara-todd]

Ah. I had tried that originally and it didn't work for me. I think in my case it's probably just that Gatsby is filtering out the environment variables available. Good to know it should work. Thanks.

[ghost]

FYI, for those who are using gatsby-plugin-netlify-cms (which is probably everyone), I don't believe the window.CMS_MANUAL_INIT = true will actually work for you. At least it didn't for me. It wasn't manually initializing for me until I set the option in the plugin. In addition to setting the config value above and calling init({ config }), you'll also need to set the manualInit flag on the plugin.

Based on what I've found, the following is all you'll have to set in order to get this working:

In gatsby-config.js:

module.exports = {
    siteMetadata: { ... },
    plugins: [
        // other plugins...
        {
            resolve: 'gatsby-plugin-netlify-cms',
            options: {
                manualInit: true, // <--- here
                modulePath: `${__dirname}/src/cms/cms.js`,
            },
        },
        // other plugins...
    ],
};

In cms.js:

import { init } from "netlify-cms-app";

window.CMS_MANUAL_INIT = true; // this doesn't do anything, at least in my testing

// You will have to do some work to set some env variable (like BRANCH)
// with the current git branch for local development. Netlify will automatically
// populate this variable for you when doing CI builds.
const { BRANCH } = process.env; 

  const config = {
    backend: {
      name: "github",
      repo: "repo/name",
      // prefer the set BRANCH if available, or fallback to `master`
      branch: BRANCH || "master" 
    }
  };

  init({ config });

For help in populating the git branch for local development, you can use a simple npm library such as git-branch

I hope that's helpful

[kiliw]

I finally have now a solution which works in my case. I hope some of you can use it too. A detailed description can be found here. Thanks @zboman and @kara-todd for the input :)

By they way I had to clear the page cache whenever I made some changes. This might be one of the basics but kept me thinking about my solution although it was already working perfectly :sweat_smile: Lesson learned :smile:

[mudlabs]

Why is this complicated?

Netlify already allows you to tie any branch to any endpoint (i.e. master to yousite.com, development to development--yoursite.com).

At the moment Netlify defaults to master branch if no branch is specified in config.yml. So why can't you just use whatever internal system exists for linking branch -> endpoint to set the default branch? Seems to me that's the only logical behaviour.

---

Random Thought: Should Netlify even be able to push to a branch that did not trigger the build?

• Developers work on a branch (i.e. feature), Netlify builds to that branch linked endpoint, developers/clients approve the build, and then developers merge into master/production.

[erezrokah]

At the moment Netlify defaults to master branch if no branch is specified in config.yml. So why can't you just use whatever internal system exists for linking branch -> endpoint to set the default branch? Seems to me that's the only logical behaviour.

Hi @mrfoster, there is no guarantee that the CMS runs on Netlify as it can be used anywhere.

Have you seen the proposed solution here?

const cms_branch = window.location.hostname.includes('develop') ? 'develop' : 'master';

const config = {
  backend: {
    name: 'github',
    branch: cms_branch,
    repo: 'owner/repo',
  },

That solution gets the branch from the URL, but you can also use env variables as described in https://github.com/netlify/netlify-cms/issues/1737#issuecomment-530992998

[mudlabs]

That requires a JavaScript config, what if you're just using a yaml config. Also anytime you (or another) developer creates a new branch that cms_branch logic needs to be adjusted. Else the new branch gets built to the master branch endpoint.

[erezrokah]

Hi @mudlabs, you can improve the logic to use a regular expression to match the branch name. If you don't want to use a JS at all (only yaml) you could write a script in your build process to find replace the branch. e.g.

backend:
  name: git-gateway
  branch: BRANCH_PLACEHOLDER

sed -i "s|BRANCH_PLACEHOLDER|${BRANCH}|g" config.yml

You can populate ${BRANCH} with whatever approach that works best for you (e.g. reading it from the git config, use process.env.HEAD if on Netlify).

[marcus-clements]

Hi @mudlabs, you can improve the logic to use a regular expression to match the branch name. If you don't want to use a JS at all (only yaml) you could write a script in your build process to find replace the branch. e.g.

backend:
  name: git-gateway
  branch: BRANCH_PLACEHOLDER

sed -i "s|BRANCH_PLACEHOLDER|${BRANCH}|g" config.yml

You can populate ${BRANCH} with whatever approach that works best for you (e.g. reading it from the git config, use process.env.HEAD if on Netlify).

This worked beautifully for me. Netlify defines an env var HEAD which has the branch name of the deployed branch. I just had to modify config.yml as above then add this line to my deploy script: sed -i "s|BRANCH_PLACEHOLDER|${HEAD}|g" content/admin/config.yml