Secrets in ecosystem.json

[dbburgess]

I'm currently working on deployments with pm2, and so far I really like how it works. There is one thing I don't get though. Generally, you keep application secrets out of your source control (api keys, database passwords, etc), and inject them into the application with environment variables. Cool, that makes sense.

The ecosystem.json file is a nice way to define the application,environments and variables to go with each for deployments. However, when I run the deploy command, I get this error: Configuration file ecosystem.json is missing. Action canceled. I presume this is because my ecosystem.json file is not on the remote server, as I haven't committed it into git.

How is this reconciled? Am I missing something? Are most people just committing their ecosystem.js and not worrying about any potential security issues with their secrets being included, or is there another way to inject the ecosystem file into the remote system? It'd be nice if the deploy command just pushed the file up or something.

[Couto]

I'm also highly interested in others's opinions about this matter, mostly because I'm also having this problem.

[robertcasanova]

Quote!

[theoephraim]

Commit your ecosystem file but store your sensitive data elsewhere (either ENV or in a file somewhere else on the machine, like the /etc folder)

I use https://github.com/indexzero/nconf to accomplish this. This example is a bit overly complex because I'm also using cson (coffee style json). But the gist is...

• load from defaults file (commited into git)
• load from locals file if exists (not committed into git)
• load from /etc file if exists (only for deployments)
• load from ENV vars if exist (currently not using them as I find the files easier, but you could)

nconf = require 'nconf'
CSON = require 'cson'
fs = require 'fs'
_ = require 'lodash'

CONFIG_FILE = '/etc/my-app-config.cson'

loadCsonConfig = (path) ->
  if fs.existsSync( path )

    # throw the error if config is bad!
    # nconf doesnt throw it so we do this first
    file_config = CSON.parseFileSync path
    if file_config instanceof Error
      throw file_config

    nconf.file
      file: path
      format:
        stringify: CSON.stringifySync
        parse: CSON.parseSync

# use nconf's in-memory store
nconf.use 'memory'

# args take precedence, then env vars
nconf.argv().env()

# Load overrides and sensitive config values
loadCsonConfig( "#{__dirname}/local.cson" ) # for dev
loadCsonConfig( CONFIG_FILE )               # for deployments

if process.env.NODE_ENV == 'test'
  loadCsonConfig( "#{__dirname}/test.cson" ) # for testing

# loads default values, dev tokens, local urls, etc
# we must load defaults last due to a bug in nconf
nconf.defaults( CSON.parseFileSync("#{__dirname}/defaults.cson") )

There are a few other modules out there that would let you accomplish something similar.

[Couto]

So that means that you environment variables are always on a file somewhere on the machine? Doesn't that defeat the purpose? I mean having the variables on the ecosystem.json somewhere on the machine, or on a .env file ends up being the same, right? (Honest question)

[theoephraim]

Maybe I'm missing something, but the important thing seems to me that your sensitive info isnt committed into version control.

To me, storing the production tokens in ENV or a file buried somewhere on the machine seem about equivalent, as long as your machine is secure (the file isnt accessible).

[Couto]

While that makes sense, it seems to me, then, that using the ecosystem.json other than a place to state hosts for deployment is pretty useless then, and that's because you need to add the ecosystem.json to git to be able to make deployments, but at the same time, you can't use it to declare env variables because those should not go into git... it's kinda conflicting...

I keep thinking that I'm missing something about this...

[dbburgess]

That makes sense to me @theoephraim, and I agree that is the important thing...But @Couto's point is why I opened this issue. I can easily find a way around this to get my configuration onto the server, but it seemed like the point behind the ecosystem.json file was to make deployments super easy, where you can just add servers and other config details, type one command, and all of the servers will receive the deployment quickly and easily.

If I have to login to each machine to update the environment variables, the ecosystem file isn't really helping me very much.

[Alex0007]

You can create private git repo with ecosystem.json and instance_key and add it to optionalDependencies

[Couto]

I'm posting this to future users, or to help with the discussion and solutions.

I had some luck with git-crypt, to encrypt the ecosystem.json and post a binary blob into the repository.

Since all developers have GPG keys, all I had to do was to create an extra key for the server (without passphrase to avoid complications and make scripting a bit easier... although that comes with a weaker security obviously) and set the post-deploy command to unlock the ecosystem.json

so my ecosystem.json deploy section becomes something like:

"development" : {
  "user" : "deploy",
  "host" : "[REDACTED URL]",
  "ref"  : "origin/develop",
  "repo" : "git@[REDACT REPOSITORY URL]",
  "path" : "/srv/[REDACTED PROJECT NAME]/api",
  "post-deploy" : "source ~/.profile && npm install && npm run db:migrate:up && git crypt unlock && pm2 startOrRestart ecosystem.json --env development"
}

I have no idea if this is most correct solution or not, but at least allows me deploy with pm2, and I don't have to leave API keys in plain sight.

[vitorbaptista]

Any news on this? How to pass environment variables to the server without simply adding them to the commited ecosystem.json?

[theoephraim]

If you really want to have your secure config values within your repo but keep them safe, you could use something like https://github.com/jed/config-leaf

[kichooo]

It would be nice to have some kind of solution for being able to use env vars with keys while not keeping them in git repo.

[yves-s]

So far I just create a ecosystem.sample.json and just clear the critical parts. After the deployment I just copy/rename the sample file to ecosystem.json and fill the important sections local and on the server by hand. Maybe not the best solution but For me the easiest and fastest.

{
      "name" : "AY-Bot",
      "script" : "dist/server/index.js",
      "env" : {
        "NODE_ENV" : "development",
        "COMMON_VARIABLE" : "true",
        "SESSION_SECRET": "",
        "WIT_ACCESS_TOKEN": "",
        "VERIFY_TOKEN": "",
        "PAGE_TOKEN": ""
      },
      "env_production" : {
        "NODE_ENV" : "production",
        "SESSION_SECRET": "",
        "WIT_ACCESS_TOKEN": "",
        "VERIFY_TOKEN": "",
        "PAGE_TOKEN": ""
      }
    }

[javimudi]

We're currently following a two-repo approach. One repo for code, and another private one just for deployment. The 'ugly' thing here is that all maintenance tasks must be launched from the deployment repo, as all running settings and env vars are stored there.

Still, having two separated repos has helped us maintaining a cleaner repo for our main app, as well as giving us the option to transform the deployment repo into a microservice, easy to scale and integrate with our CI tools.

[jaimeagudo]

Hi guys, I am concerned with the matter as well. Thanks for sharing to everyone.

Thinking of using http://pm2.keymetrics.io/docs/tutorials/capistrano-like-deployments but haven't much time to explore, if someone has achieved something tidy please share 👍

@theoephraim approach seems too complicated for my simple mind 😅 , @vitorbaptista is simpler but I don't wanna mess up with encription.

For now what I do is to have dedicated "deploy" (like @javimudi) repo and do stuff like this: (just assuming the deploy repo is cloned and up to date on the deployment machine ofc, you might add a git pull anyway)

"post-deploy": "ln -s ~/git/infraestructure/parse/* .; source ./mongo_auth.sh; npm install; pm2 startOrRestart ecosystem.json --env production",

On my development machine I have a ln -s ../infraestructure/ecosystem.json . as well.

[jamesjessian]

I found this thread after reading about the Serverless Framework and how that allows you to keep variables ready for deployment in an encrypted file. You provide the password to that encrypted file as a parameter to the serverless deploy command. I thought this would be nice to have when launching our apps with PM2.

[mbilokonsky]

Also running into this exact question. It seems weird to me that the config file couldn't be copied up when you run a deploy setup, via scp or something?

Like, it's a whole deployment orchestration tool, I'm confused why there's not a way to deploy secrets without committing to github, made me feel like I'm just missing something. Glad to see I'm not alone?

[mbilokonsky]

Honestly even environment variables aside, this means I have to commit a file to my repo with my server info...? This seems super weird to me, I think I am not gonna use it.

[elrumordelaluz]

Agree that could be absolutely easier to have a way to avoid commit the fileecosystem with its secrets to the version control.

For now I solved with a separated file in another location in the server and calling it when post-deploy .

[rafaelangarita]

Is there a solution to this? I too thought my deployment was going to be painless until I got the 'ecosystem.config.js not found' error.

My workaround is to scp my ecosystem files before 'pm2 deploy'. It's kind of awkward tough., I feel it would be the same to scp the files and ssh into the server to do the deployment from there.

[s0l0ist]

A good solution I've found is to combine pre-deploy-local and post-deploy to copy over the full ecosystem file (with secrets) and then remove it afterward:

... "pre-deploy-local" : "scp ecosystem.json user@server:'/home/node' ", "post-deploy" : "git checkout dev && npm install --silent && pm2 startOrRestart ~/ecosystem.json --env development && rm ~/ecosystem.json" ...

While this isn't ideal, it works for now until someone else comes up with a more pure way.

[DiegoRBaquero]

Do it like this:

'post-deploy': 'source .env && npm install --production && pm2 reload ecosystem.config.js --env production --update-env'

And put your vars in .env with the format: VAR=VALUE

[nfantone]

If using something like nconf:

• You can specify a config file to parse through command line args or env vars.

  nconf.argv()
   .env()
   .file({ file: nconf.get('config') });

# And start it like:
node app.js --config /etc/my-app/config.json

• Encrypt your configuration.
• Export environment variables from a file local to your deploy server, just like what was mentioned above.
• Use a remote config repository/datasource like redis or etcd.

[skylarmb]

The easiest solution I came up with is to use dotenv (https://github.com/motdotla/dotenv/).

1. Store a file named .env somewhere on your server containing your env variables. Edit or deploy this file however you like, the point being it's not part of your repo or ecosystem.json.
2. Add a step to your post-deploy command in ecosystem.json to copy this file into the current deployment folder. Here is mine:

...
"post-deploy": "npm install; cp ~/.env .env; pm2 startOrRestart ecosystem.json" 
...

3. At the top of your main app file, require dotenv with a relative path

const path = require('path')
require('dotenv').config({path: path.join(__dirname, '.env')})

and boom, all your variables defined in your .env file are now available in process.env

[rohit-gohri]

@skylarmb I'm doing something similar to this.

Just skipping the step of having to make an env file in the server, by using ssh to copy the local env file to the remote host in the pre-deploy-local command.

Something like : scp ~/.env ${user}@${host}:~/.env

Or use rsync.

But it would be nice if pm2 had some way to copy different secrets files depending on different hosts and envs. I have a workaround for that, but it complicates the ecosystem file with too much logic.

[skylarmb]

@rohit-smpx I like that solution! In our case the .env file does not exist on the machine we deploy from (a transient container in a CI/CD system), so having the file persist on the server is still best for us. If you are deploying from a machine that has access to the .env file your solution is definitely ideal

[danielo515]

This is ridiculous. If I have to copy the file myself before launching the pm2 command I can just log into the server and send the pm2 command there. Without this feature, pm2 deploy is absolutely useless. Some people has suggested some smart solutions by using scp on the pre-deploy hook. This is nice if you want to copy the file to the remote server, but that is not my case. To be honest, I don't understand why this is as it is, it is a limitation. If PM2 is able to run commands on a remote machine (which is basically what this deploy thing is all about) I don't understand why it can't just read the env variables locally, and launch a command with those env variables set as if you were typing them. Something like this:

NODE_ENV=production SECRET_KEY=asdffsdfsd pm2 start

Is that impossible?

[skylarmb]

@danielo515 I am not an expert on pm2, but what you are suggesting is that you pass all your local environment variables to the remote machine when you run a pm2 command? That seems like recipe for complete disaster and is probably not what you actually want if you really think about it. Do you really have your production environment variables set locally (you shouldn't)? Are you deploying from a Mac to a linux server? If so your environment will be wildly different and incompatible. This will also pass a ton of garbage env variables that the server doesn't care about. This would be a complete disaster.

Having your environment variable management decoupled from your process / log manager is a good thing. For example the machine that initiates the pm2 command may not have access to the appropriate environment variables in order to pass to the server (like in a CI/CD pipeline or deploying from a local machine). Also the way environment variables are managed is highly dependent on the setup of the application and infrastructure. Are you using Chef / Puppet? How are your env variables stored? Are your env variables in your Docker image? In an encrypted git repo or a KMS? In a third party tool? How are permissions granted for access to environment variables? etc.

I think overall the decision to have pm2 by default not be responsible for setting any environment variables encourages proper application architecture and doesn't break existing deployment processes. Whatever method you use for managing environment variables, just use pre-deploy hooks to set them up before pm2 runs. If you don't have an actual method of managing your environment variables, then persisting the .env file on the server or using scp like @rohit-smpx did is your easiest option.

Also, in your own words "I don't understand why it can't just read the env variables locally, and launch a command with those env variables set" You just described the scp method. scp would "read the env variables locally" (from a file) and then "launch a command with those env variables set" (once you run your application that uses dotenv or similar library they will be available in process.env).

[danielo515]

@danielo515 I am not an expert on pm2, but what you are suggesting is that you pass all your local environment variables to the remote machine when you run a pm2 command? That seems like recipe for complete disaster and is probably not what you actually want if you really think about it. Do you really have your production environment variables set locally (you shouldn't)?

Nope, that is not what I'm saying. Let me explain it with an example. Imagine an ecosystem.json like this:

module.exports = {
  apps: [{
    name: "myApp",
    script: "src/start.js",
    env: {
      PORT: 8080,
    }
  }]
}

then pm2 can read this file and execute the following on the server:

PORT=8080 node src/start.js

So my suggestion is to just use the variables defined on the ecosystem.json file and set them explicitly. I don't think this is such a big deal, it does not have all the garbage of multi env problems that you are mentioning. Probably there are way better ways of doing this, but for simple cases having this as an extra option will save someone's day.

Also the way environment variables are managed is highly dependent on the setup of the application and infrastructure.

O sure, but in my case I'm using just PM2, and I don't want to use anything else

[nfantone]

@danielo515 I'm not entirely sure what you are asking for here but you can already define env vars before starting an application. Your "hypothetical" ecosystem file would work just fine.

You can even define them by deploy environment. Here's an example configuration.

{
  "apps": [
    {
      "name": "my-api",
      "script": "app.js",
      "max_memory_restart": "512M",
      "exp_backoff_restart_delay": 100,
      "wait_ready": true
    }
  ],
  "deploy": {
    "production": {
      "user": "myuser",
      "host": "prod.myapi.com",
      "ref": "origin/master",
      "repo": "git@gitlab.intra.net:group/my-api.git",
      "path": "/opt/group/my-api",
      "post-deploy": "npm ci --production && pm2 reload deployment.json --env production",
      "env": {
        "NODE_ENV": "production",
        "PORT": "8080",
        "DB_URI": "mssql://prod.db.uri:7878/db"
      }
    },
    "qa": {
      "user": "myuser",
      "host": "qa.myapi.com",
      "ref": "origin/develop",
      "repo": "git@gitlab.intra.net:group/my-api.git",
      "path": "/opt/group/my-api",
      "post-deploy": "npm ci --production && pm2 reload deployment.json --update-env --env qa",
      "env": {
        "NODE_ENV": "qa",
        "PORT": "8070",
        "DB_URI": "mssql://localhost:1234/db"
      }
    }
  }
}

[danielo515]

thanks, @nfantone, that is exactly what I was looking for! Now, this is not an issue anymore for me, thanks again.

[imkane]

I've read about security issues with having a .env file on your production server, with 3rd party packages gaining access to it.

Also, what about nefarious server admins or developers seeing all your secret/private keys? I'm not sure the best way to handle this. I've been Googling all morning too.

Any thoughts on this stuff?

[kavishatalsania]

@nfantone According to your solution, we need to push deployment.json file into git repository. Right?

[nfantone]

@kavishatalsania Not really, no. My answer was directly related to @danielo515 problem and not addressing security concerns - or anything but the topic in question, to be honest.

In terms of your own question, what I would typically do, depending on the project's complexity and requirements, is set a config file with default values and push it along with everything else to the repository (defaults would typically point to a "local" or "dev" environment, aimed at enhancing DX and nothing else) - then either:

1) Have a config file per environment, locally in your server. You don't check in this file into your repository, you rely on it existing under some known path (like /etc/yourapp/app.conf).

This has the disadvantage that @imkane pointed out regarding "malicious" admins, developers and third-party modules being able to read your config. It's a very valid concern, but to be honest, this also applies to anything else. If you encrypt your settings and store "secrets", whoever/whatever holds the super-secret-key can pretty much manipulate your config the way they want. We rely on so much third-party, open source software nowadays, that these vulnerabilities are really hard to control. This is specially true for JS projects.

No solution is perfect: your security is only as good as the people handling it. If you don't trust your own servers, developers and admins, I'd argue your issue lies somewhere else.

This approach could also lead you to have redundant configuration files, depending on your infrastructure - which could bring out inconsistencies. If you, say, deploy your app into multiples nodes behind a load-balancer, you would need to ensure that each node has a copy of your config file available to them. There are, of course, multiple ways to mitigate this.

2) Have some sort of external config repository. There are many known services that handle this pretty well: etcd, ZooKeeper, AWS SSM Keystore or even Redis. You then inject these configuration dependencies when starting the application (if they are runtime config values) or building it in whatever your deployment/build pipeline is.

If you are using some kind of serverless or clustered setup (AWS, GCP, Azure, Kubernetes, et al.), chances are you already have one these systems at your disposal.

[kavishatalsania]

@nfantone Thank you for your response. Yeah, I can create config file per environment locally in my server and use some external config repository. But is there no way PM2 read secrets from my local ecosystem.config.js file before deploying project and read from there?

[mattdeluco]

Would it possible to do something like $ PM2_ENV='SECRET="xyzzy"; OTHER_SECRET="quux"' pm2 deploy production?

As someone mentioned here, one wouldn't want pm2 to copy the entire environment from the local machine to the remote machine on deployment. So instead, I propose a specific pm2 environment variable for the local machine which contains a list of environment variables to be set and used upon running pm2 on the remote machine.

[hugomallet]

Same issue with pm2-runtime. I managed to override env vars from the command line using ecosystem.config.js:

const {
    LOG_LEVEL = 'info',
} = process.env;

module.exports = [
    {
        script: 'server.js',
        env: {
            LOG_LEVEL,
        },
    },
];

Then in the command line I can override with env vars (or dotenv):

LOG_LEVEL=trace yarn pm2-runtime ecosystem.config.js

[LouisVA]

thanks, @nfantone, that is exactly what I was looking for! Now, this is not an issue anymore for me, thanks again.

Hey, this might be a bit random because I am in a similar situation. This worked for you in the sense that you put your keys in your config file (in the git/vcs) and it deployed using that. Or that you managed to inject the keys in your config (on the computer you are using to deploy something remotely) into the remote deployment build? I'm trying to do the latter and well no success.

[danielo515]

thanks, @nfantone, that is exactly what I was looking for! Now, this is not an issue anymore for me, thanks again.

Hey, this might be a bit random because I am in a similar situation. This worked for you in the sense that you put your keys in your config file (in the git/vcs) and it deployed using that. Or that you managed to inject the keys in your config (on the computer you are using to deploy something remotely) into the remote deployment build? I'm trying to do the latter and well no success.

The first one. I just wanted to not have to put the env variables on the target machine, or a config file on the target machine. What I would do nowadays is either:

• Use ansible to deploy
• Use a template to generate the config before deploying with pm2 (you can also use ansible for this)

[nfantone]

@danielo515 is on the right track. An even simpler approach that I have used in the past is, if you trust your app server and users, just place something like an .env file in your project's root directory and enable/read it after deploying.

This obviously doesn't scale, but for simple setups it solves the problem without hassle.

[LouisVA]

@danielo515 thanks for mentioning ansible, didn't know about this technology @nfantone I'll most likely do a temporary fetch and delete instead of keeping the file. I also see some justification in a repo with encrypted blobs (John Resig brought this up in a blog) until it's worth setting something up like Jenkins.

[noemission]

On the same issue, no matter how we encrypt our passwords and keys if we end up passing them with env variables, everything is written in plain text at ~/.pm2/dump.pm2. Is there any way to truly hide all this sensitive information?

[danielo515]

On the same issue, no matter how we encrypt our passwords and keys if we end up passing them with env variables, everything is written in plain text at ~/.pm2/dump.pm2. Is there any way to truly hide all this sensitive information?

Well, at the end the information needs to live in plain text at some place. I mean, sure you can encrypt it, but then where do you store the decrypt key? In plain text somewhere, or in memory somewhere. The correct thing to do is to set proper permissions of that file, so only admins can read it. The only way to not have the info at some place is to start the process manually providing the required credentials as options and make sure the command is not registered in bash history.

[AlbertoFabbri93]

This definitely complicates the deployment, it would be very nice to have this functionality built-in without having to use .env files.

[kumail-raza]

after going through the discussion above the conclusion is instead of using pm2 save , for the time being the workaround could be as follows:

on Linux cron, on Windows scheduler/service to run the command manually when the system restarts pm2 start ecosystem.js --env prod

a quick fix should be made instead of storing the secrets as plain text in ~/.pm2/dump.pm2 reading from .env file, a way to provide the envFile