Simplify environments complexity

[levibostian]

There are a few problems that group together that I would like to fix.

These issues all have something in common. Refer to this article for inspiration behind how we fix these issues. We need to use environment variables for our application to run.

1. I have many docker files and are all very similar. Let's take the docker/db/ files as an example. I have a base docker-compose.yml file, then override files for staging, production, testing, development, etc. This should just be 1 file that runs the same everywhere. That way, when we run in dev and testing modes, it's similar so we can feel better about production working.

You should be able to configure the docker compose files by just setting environment variables.

We should not need to have all of these docker compose files, just 1 or 2 files. Also, the package.json scripts should be simplified to just have 1 script for starting a database, for example.

Sequelize CLI can be simplified here as well. We don't need to generate a config file for the CLI to read, we just need to dynamically change the database to run the CLI against by environment variables.

2. We dynamically enable and disable code in our app's source code depending on the environment that we are running.

  if (isRunningDatabase) {
    await initDatabase() 
  }

  if (!isRunningOnProduction || enableLogging) {
    let extraInfo = extra ? JSON.stringify(extra) : "(none)"
    console.debug(`DEBUG: Extra: ${extraInfo}, message: ${message}`)
  }

I don't think this should be the case. All the code should run, all the time. This is bad because at production, we cannot feel confident that our code works. Honeybadger, postmark, firebase are all examples. We should think about creating development and staging versions of postmark and firebase accounts. Setup honeybadger to not upload in certain environments, but still configure it. That way when production runs, we can feel confident it all works.

So, the code base should be setup in a way that says, "This code runs, no matter the environment. You just need to set system environment variables accordingly and I will work. If I run successfully in 1 environment, I will run successfully in all. You can just configure me differently with diff api keys and all with environment variables".

The code base should not need to say, "Hey, run this way in this environment, but run this way in another environment". No need. It should run the same in all.

3. Everyone has the ability to set their own environment variables. There is not a "base" or "development" environment setup for development purposes. What if someone wants to make their own configuration?

Thinking, can use an optional tool on your local development machine and run like this:

$> set_environment --name project-name

Then, it looks in ~/.set_environments/project-name on your machine and will set those environment variables on your machine. So, when you run your application, it will use those environment variables. This allows anyone to set their own values on their machine and be able to work on multiple projects at one time.

4. Travis is where we store all prod and staging variables. No where else. It's what packages up the application for shipping for us. This helps us to make sure that we don't put secrets in our source code by accident. The CI should store all secrets for us.

5. When environment variables are needed to be added to our application, somehow, we get reminded to add it to your system.

So, we can do this:

Create a file like .required_env.json that has a list of environment variables that your entire app requires.

{
  "DATABASE_USERNAME": {
    "description: "Used to connect to the database"
  }
}

Then, we can have a tool (start withthis?) that checks this file and then checks with your system to make sure that it has that variable set.

Then, in our npm scripts, we can run this program:

"start": "npx assert_env && npx forever app.js"

It will throw an error if you did not set a variable on your system:

$> npm run start
Error: You forgot to set `DATBASE_USERNAME` - Used to connect to the database.

---

In conclusion, by simplifying our environments, we remove lots of complexity and can feel more confident our application will run in production.

[levibostian]

This PR is currently failing on trying to test the production docker container builds and runs successfully.

It fails because the container builds and runs, but the application does not begin successfully because sequelize cannot connect to the production database. This is because I have sequelize using a config file:

import { Dialect } from "sequelize"
export interface DatabaseConnection {
  username: string
  password: string
  database: string
  host: string
  port: number
  dialect: Dialect
}

const testConnection: DatabaseConnection = {
  username: process.env.POSTGRES_USER!,
  password: process.env.POSTGRES_PASSWORD!,
  database: process.env.POSTGRES_DB!,
  host: "localhost",
  port: 5433,
  dialect: "postgres"
}

const developmentConnection: DatabaseConnection = {
  username: process.env.POSTGRES_USER!,
  password: process.env.POSTGRES_PASSWORD!,
  database: process.env.POSTGRES_DB!,
  host: "localhost",
  port: 5432,
  dialect: "postgres"
}

const stagingConnection: DatabaseConnection = {
  username: process.env.POSTGRES_USER!,
  password: process.env.POSTGRES_PASSWORD!,
  database: process.env.POSTGRES_DB!,
  host: process.env.DATABASE_HOST!,
  port: 5432,
  dialect: "postgres"
}

const productionConnection: DatabaseConnection = {
  username: process.env.POSTGRES_USER!,
  password: process.env.POSTGRES_PASSWORD!,
  database: process.env.POSTGRES_DB!,
  host: process.env.DATABASE_HOST!,
  port: 5432,
  dialect: "postgres"
}

const connections: {
  development: DatabaseConnection
  production: DatabaseConnection
  staging: DatabaseConnection
  test: DatabaseConnection
} = {
  development: developmentConnection,
  production: productionConnection,
  staging: stagingConnection,
  test: testConnection
}

// Exists for sequelize-cli to find the connections as the main export.
// Note: Must have the export default be the same object as the module.exports or, module.exports will export erase all other exports.
export default connections
module.exports = connections

And then depending on the NODE_ENV variable, it will pick a connection. Well, we have the production environments hard coded in .env.production currently. This is not good because I need to edit the DATABASE_HOST for this test. I need to set it to localhost, but in my running app, I need it to be set to the IP address of where the application is running.

So, with the fix of this issue, this PR can be fixed. I can change the environment variables to satisfy the needs to pass this test.

[levibostian]

As for firebase-admin node module, this article will help in how to set this up. I believe we can use environment variables to popular all of these values as well.

[levibostian]

This is done. 1 .env file will setup the whole config now. I only have 1 dockerfile for the app and 1 for db. All services run all the time (no matter the env) so we can have close parity between all of the environments.