Introduce project-level environment variables

[gtsiolis]

Problem to solve

The product currently supports only user-level environment variables set in user settings, where you use a scope syntax to specify for which projects the variable should be available to use. See /variables and relevant docs.

With the upcoming Teams & Projects features, it will be able to set up a project and share this project configuration with other team members under a team.

Proposal

Given there's also a plan to add project and team settings, it could make sense to add support for project-level environment variables or even team-level environment variables. This could also protect variables from being accidentally deleted or changed, and more as these could be read-only to non-owner members.

TBD (To be discussed)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ntziolis]

Any updates?

[gtsiolis]

Hey @ntziolis! FYI, We've added this feature request on the roadmap[1] for the next quarter. 🗺️

[ntziolis]

Great to hear. This is currently blocking us from adopting the gitpod way of doing things.

Since we cannot use prebuilds (private npm registry), each dev currently creates a single pinned workspace per project and reuses it. This is because creating a new workspace per issue would take too long without prebuilds.

[shaal]

+1 for this feature. @gtsiolis Adding a team member (and giving them access to secret), happens through access to the repo on Github/Gitlab/Bitbucket? or I need to add a team member to the repo, and then additional steps in the Projects interface inside Gitpod website?

[ntziolis]

I would prefer to keep gitpod teams and github teams separate. We have third party devs that are working on code that doesn't need access to the secrets and we would like them to continue to not have access to them.

That said, I could see this being solved by

• making this auto-add configurable (meaning one can turn it off when this is not wanted)
• extending the gitpod browser extension that adds an additional button to the add team memeber section on github/gitlab etc (add to github and gitpod team)
• and/or via a notification to the gitpod admin whenever there is a new github team member added so he can take action

[AlCalzone]

It would already be immensely helpful to be able to define (public) env variables in .gitpod.yml without any UI whatsover.

Our repo is using yarn to manage dependencies. The repo is configured to use a global yarn cache, but this does not work with gitpod prebuilds because only /workspace is persisted.

So we're changing the yarn config in the prebuilds to use a local (persisted) cache, but without extra steps does not carry over to workspaces spawned from the prebuilds. To avoid changing the config files, we need to set environment variables for each workspace. This can be done with startup scripts, but is currently very repetitive. With two startup tasks, we need to do this:

tasks:
  - name: "Task 1"
    before: |
      # set env variables for this terminal
      export YARN_CACHE_FOLDER=".yarn/cache"
      export YARN_ENABLE_GLOBAL_CACHE="false"

      # Persist the env variables for other terminals
      echo "export YARN_CACHE_FOLDER=\".yarn/cache\"" >> ~/.bashrc
      echo "export YARN_ENABLE_GLOBAL_CACHE=\"false\"" >> ~/.bashrc
    # ... rest of task 1

  - name: "Task 2"
    openMode: split-right
    before: |
      # set env variables for this terminal
      export YARN_CACHE_FOLDER=".yarn/cache"
      export YARN_ENABLE_GLOBAL_CACHE="false"
    # ... rest of task 2

[SilvanCodes]

Hey @ntziolis! FYI, We've added this feature request on the roadmap[1] for the next quarter. :world_map:

Looking forward to see this land! 👍🏼

[louis030195]

+1 for this feature 👍

[flauc]

+1

[andreafalzetti]

I can't wait to have this feature 👀 I need to run prebuilds and be able to install private NPM packages

[XBeg9]

meanwhile, what will be the recommendation to use a private registry for prebuilds? is env vars injected into custom docker build phase?

[shaal]

@andreafalzetti @XBeg9 I am using my user environment variables during prebuild here - https://github.com/shaal/DrupalPod/blob/main/.gitpod/drupal/envs-prep/prepare-environments-gcs.sh#L8

[XBeg9]

@shaal does it mean that every single user should put this env into config until this feature will be done?

Just out of curiosity, when prebuild triggers, specifically for team, which envs will be injected into the build pipeline?

[ntziolis]

@shaal Are you sure this works? @gtsiolis Does prebuild have access to user env variables? If so which user? the user that connected the repo? This could be a great workaround if thats the case. So far and based on my tests I could not confirm this.

[XBeg9]

@ntziolis same here, if envs are injected into prebuild phase even from project / team level, this is gonna be a security issue

[jgallucci32]

As a workaround try creating the file .gitpod/Dockerfile then specifying inside of it

FROM gitpod/workspace-full:latest
ENV YOURENV=yourvalue

Then in your .gitpod.yml file you would add this entry

image:
  file: .gitpod/Dockerfile

This will allow the env vars to be configured per-project and stored in source code for the project.

[XBeg9]

this way you are exposing the secret, right? @jgallucci32

[jgallucci32]

If you had sensitive data in the env var then yes. But the examples above were using all non-sensitive env vars so this workaround would be OK in those cases.

[shaal]

@XBeg9 @ntziolis Yes, this works 100%.

But I did not try this with a team. When I manually run prebuild, or when I push a commit and prebuild runs automatically, it's reading the environment variables that are saved in my Gitpod user account.

If it couldn't read the environment variables, it wouldn't be able access the Google Cloud Storage.

[ntziolis]

When I manually run prebuild

@shaal Interesting, I wasn't aware that running prebuilds manually was possible. Turns out this seems to be due to us using github. The gitpod prebuild registration process for gitlab/gitbucket seems to include an additional step of running the initial prebuild manually to register the necessary hooks, I found this info here.

I think that's very interesting by itself as I would have expected prebuilds to always run user agnostic.

Follow-up question (now purely out of investigative interest): Will your env variables be also used for any automatically triggered prebuilds?

If it doesn't work for automatic prebuilds it would kind of void the purpose of prebuids which is to reduce the startup time when you open a workspace. Unless you manually trigger a prebuild after every push. Which could be a workaround until this feature lands.

@rl-gitpod Is there a way to trigger manual prebuilds for github that leverage the user context (env variables) of the user that triggered the perbuild?

[shaal]

@ntziolis yes, when I push commit to a branch, it automatically triggers a prebuild run (which is set at bottom of .gitpod.yml), and it can read my user variables. I do not know how it does it, or what will happen when someone else will try running it (I would guess and hope it will look into who's the user that pushed the commit, or triggered the manual prebuild)

[geropl]

Extennded feature request: Add "Team Environment Variables" to prebuilds

In the past we would pass personal environment variables into all prebuilds. This enabled usecases where you need to integrate with third-party services that are not public (log infra, build caches, docker registries, ...). IMO it's important that we: a) provide this extension point b) be consistent between prebuilds and workspaces, to make sure builds behave the same everywhere

We disabled that in the past due to concerns about implicit exposure of secrets with the introduction of Teams&Projects (or even slightly before). With environment variables on Team Level it's clear that those are shared within the team, so this concern is no longer valid. :rocket:

/cc @jldec @JanKoehnlein

[XBeg9]

@geropl I agree, it doesn't make any sense to inject user-level ENV variables into the prebuilt phase, this way you can expose a lot of sensitive information. Project / Team level is a way to go.

[ajimix]

Came here after hours wondering why the environment variables were not exposed on the prebuild. I wish gitpod had documented it in their docs...

In any case, looks this is a big stopper for our company as we use private repositories which require private keys and we cannot expose them into the source code. I hope this can be implemented soon as it's a stopper for any medium/big company. Not using prebuilds is not possible as would make the workspaces very slow to start for the team

[jldec]

For the folks on this thread (thanks for all your feedback b.t.w.) I was wondering how we might get this done incrementally.

One possiblity as a first step may be to add support for "non-shared" or "personal" prebuilds which get access to your personal env vars.

Personal prebuilds would be auto-triggered on commit, but only for the owner-user. Other users would trigger those prebuilds manually or on workspace start. Each user would have to configure the secrets required by the prebuild in their own environment settings.

This is just an idea, but something similar to this will also be required when we implement project-level env vars as discussed here. Prebuilds for projects are not currently restricted to project-team members, but you do want to restrict prebuilds with project-level secrets to just the team members who can see those projects.

[ntziolis]

Manually triggering prebuilds negates the usefulness of prebuilds.

Number one complaint of our devs when using gitpod is startup time.

Whatever the interim solution would be, it would have to allow for prebuilding automatically for commits made by anyone.

Options:

• always prebuilds with the env variables of the user that pushes a commit
• allow to specify a user per project and then always use his env variables during prebuild only (and only during prebuild)

I would tend to prefer the first option. If there are security concerns on env variables leaking:

• only use variables that are scoped to specific projects instead of /
• or ask user on first push if his variables should be shared if not no prebuild

[shaal]

@ntziolis

Manually triggering prebuilds negates the usefulness of prebuilds.

I think it depends what happens during prebuild, in DrupalPod, manual prebuilds make perfect sense because it's a repo to prepare Gitpod workspace that host additional projects.

Number one complaint of our devs when using gitpod is startup time.

I'm curious to know what is the startup time you are getting with Gitpod workspace, and what is a "good" startup time for your team. I've been working on lots of optimization in Gitpod+prebuilds and I would love to share with you my findings, we can discuss on Gitpod's Discord (my username there is shaal)

[ntziolis]

Manually triggering prebuilds negates the usefulness of prebuilds.

I should have said in many use cases vs such a blanket statement.

I'm curious to know what is the startup time you are getting with Gitpod workspace, and what is a "good" startup time for your team.

We are seeing ~3-5min wait time without prebuilds. This is def too long to embrace the gitpod philosophy of always operating in a new workspace per task. Which we really would love to embrace but cannot.

Where does our wait time come from

1. We do node development using use nx for workspace management
   To install the sheer amount of dependencies already takes some time but is much faster than locally for sure (but then again locally you don't really do it fresh for each task). However what hurts time the most is the ngcc process. Especially for the professional UI component frameworks which are usually 90+ components

   • angular
   • often capacitor + ionic
   • and almost in all projects the customer requests to leverage a specific professional UI component framework (think kendo, devexpress etc.)

2. We leverage containers for our backend infrastructure using docker-compose to start them

   • This involves building containers as well as then deploying them
   • Since they are interdependent this can only happen in a specific order (postgres + hasura + postgres web admin)

I want to be clear:

• Starting a workspace on gitpod is 2x+ faster then setting up the same project locally the first time on most of our dev machines.
• This is not because these machines are weak, but because gitpod is just blazingly fast
• The issue arises because we wane move to the gitpod way of doing things
• This means especially embracing starting fresh on every task. All devs agree that this would bring huge benefits. But also agree 3-5min is too long for this to be feasible.
• Again, to be clear this way of working would not even be an option locally

I hope this was helpful context. If you need more detailed info I'm happy to provide as much as you need here or jump on a quick call.

[ntziolis]

@shaal I just saw your reference to discord hopping on there now.

[JanKoehnlein]

/schedule

[jankeromnes]

/assign

[jankeromnes]

Okay, we've decided to move forward with Project-level environment variables, and I'll build a first iteration.

We'll likely implement this as a new feature under a Project's settings, and worry about aligning or deprecating the current User-level variables later. (This way, we can make it 100% clear upfront where/when exactly the new Project-level variables are exposed, and there is no overlap with the current User-level variables, thus we avoid unexpected/surprise variable leaks by design.)

Type	Defined In	In Prebuilds?	In Workspaces?
User-level env vars	/variables	No	In user's workspaces
Project-level env vars	/:team/:project/variables	In all :project prebuilds	In all :project workspaces

Next up, I'll try to clarify the full Project-level env var access matrix, in order to make sure that we cover all desired use cases safely. (Spoilers: Both Gitpod Team membership and GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables. This is likely okay for private repositories, but raises questions for public repositories, for which anyone can consume prebuilds.)

[shaal]

@jankeromnes Thank you for all the work on this! Did I understand your last comment correctly (ie: GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables) ?

If I have an open-source project, hosted on Github, that will use Gitpod's future project-level environment variables, anyone that open that project in Gitpod, will have access to the project's environment variables?

If that is true, would you consider changing it to give the access to project's environment variables, only to people with "write" access to that repo? (so also Prebuilds can run, and in my case with DrupalPod, access Google Cloud Storage with secret keys)

[ntziolis]

Thank you for the update. I would suggest to make the following disabled by default.

GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables

And i agree with @shaal, that if there is such implicit access to secrets it should generally be limited to folks that have a certain trust level. Write access to the source seems like the right path here.

[jankeromnes]

Thanks @shaal and @ntziolis for your feedback!

Did I understand your last comment correctly [...] If I have an open-source project, hosted on Github, that will use Gitpod's future project-level environment variables, anyone that open that project in Gitpod, will have access to the project's environment variables?

Yes, this is the current idea, but I'm happy to further discuss it so that we can work towards a better solution that addresses all use cases safely.

The arguments going for "project variables available in prebuild are also available in (any) workspace" go as follows:

• We want Prebuilds and Workspaces to remain consistent for the same Project/repository, i.e. your init tasks should give the same result in a Prebuild or in an interactive Workspace. However, if init relies on a variable that's present in Prebuild but not in Workspace, it might surprisingly fail in a Workspace.
• Prebuilds of public repositories are inherently accessible to the public (as currently designed). If you prebuild a public repo, anyone opening the same commit will get that prebuild. If the prebuild contains sensitive data (e.g. secret environment variables, or cached secrets from tools that authenticate via env vars, etc) you might accidentally "leak" them even if we explicitly don't include env variables from Prebuilds in Workspaces.

(so also Prebuilds can run, and in my case with DrupalPod, access Google Cloud Storage with secret keys)

I understand that "provide environment variables to prebuilds of a public repository without sharing them with users of that public repository" is a use case that isn't covered by our current design. (Note: this limitation only applies to public repositories -- for private repositories, you can only use them in Gitpod if you already have access to them, so environment variables are effectively only accessible to project members.)

Currently, we're here:

Env Variable Type	Team can read?	Team can write?	In Prebuilds?	In Workspaces?
User-level	No	No	No	Only for user
Project-level	No?	Yes	Yes	Yes?

However, thinking again, maybe "In Workspace?" could actually be a configuration option per Project-level environment variable. E.g. a checkbox when you create a new variable:

New Environment Variable for {Project}
--------------------------------------
Name: _________
Value: _________
Visible in Prebuilds: [x] (always checked + disabled)
Visible in Workspaces: [ ] (configurable)

But for consistency, if we allow hiding some Project variables from Workspaces, we also need to do a good job of protecting the value from "leaking" outside of Prebuilds. This means doing as good a job at it as other CI services, e.g. by doing:

• Prebuild log censoring (e.g. if you accidentally print your secret in the logs, we could automatically censor it before the logs get sent/displayed anywhere)
• Limiting who can trigger new Prebuilds for your Project (e.g. you may want to disable Prebuilds for pullRequestsFromForks -- luckily, that's already the default, but we'll need to verify it's indeed the default for all Git providers, and also make the security implications of enabling this very clear & visible)
• Or, more drastically, maybe we could allow restricting Prebuilds to only Team members... but this starts undoing the primary benefit of Prebuilds, which is making the workspace initialization faster for everyone

[jankeromnes]

However, thinking again, maybe "In Workspace?" could actually be a configuration option per Project-level environment variable. E.g. a checkbox when you create a new variable:

New Environment Variable for {Project}
--------------------------------------
Name: _________
Value: _________
Visible in Prebuilds: [x] (always checked + disabled)
Visible in Workspaces: [ ] (configurable)

Alternatively, a more fine-grained option than a checkbox to control the visibility of Project environment variables could be a visibility selector with options such as:

Who can access the value of this environment variable?
- Noone (accessible during Prebuilds only, censored out from Workspaces and Prebuild logs)
- Everyone at <TEAM> (accessible during Prebuilds and in Team Workspaces, censored out in non-Team Workspaces and Prebuild logs)
- Everyone on the Internet (accessible during Prebuilds, in all Workspaces, and visible in Prebuild logs)

However, for simplicity's sake, let's keep it as a on/off checkbox during initial implementation. This option could be a follow-up enhancement in case of strong interest.

[jankeromnes]

Update: This feature is now deployed! 🎉

Here is how to use it:

1. Navigate to your Project in Gitpod's dashboard
2. Go to Settings
3. Click on Variables in the left menu

The UI allows to define new environment variables that will be visible during your Project's Prebuilds. 📝

[jankeromnes]

Also FYI, I'll give a short demo of this feature in today's open office hours on Discord, in just under 1 hour. 🙂

[devxpy]

Can this be done via the cli? The UI is too hard to use for 20 env vars

[axonasif]

Hi @devxpy, not yet but could you please react with a " 👍 " on the following issue?

• https://github.com/gitpod-io/gitpod/issues/12208

If the CLI support was there for projects, you could use such a method to quickly add env vars in bulk.

• https://github.com/gitpod-io/gitpod/issues/12243#issuecomment-1221755010, and this of course is another feature request to have a built-in command instead of doing it ourselves.