Add support for production environment file

[t-book]

When starting geonode-project within docker, environment variables are populated from .env. In case somebody starts the stack for development and later in production everytime the .env file has to be updated accordingly. This ticket should discuss if the following can be improved (and hence added to the repository).

1) add an empty .env-production to the repository (and decide if it will by ignored by git or not) 2) add docker-compose-production.yml to the repository with following content. (.env_production will be added after .env from docker_compose.yml

version: '2.2'
services:

  db:
    env_file:
        - .env_production

  geoserver:
    env_file:
        - .env_production

  django:
    env_file:
        - .env_production

  celery:
    env_file:
        - .env_production

  geonode:
    env_file:
        - .env_production

  letsencrypt:
    env_file:
        - .env_production

4. In case of going live one can leave the .env untouched and fill the .env_production:

SITEURL=http://example.com
GEOSERVER_WEB_UI_LOCATION=http://example.com/geoserver/
GEOSERVER_PUBLIC_LOCATION=http://example.com/geoserver/
GEONODE_LB_HOST_IP=http://example.com
HTTP_HOST=http://example.com
ACCOUNT_OPEN_SIGNUP=False

5. start the stack with: docker-compose -f docker-compose.yml -f docker-compose-production.yml up -d

[gannebamm]

I like this and it actually is the way we use docker-compose and .env files for production and development. The open question is how the hardening operations (like encrypted password for GeoServer) will get performed? See SPC with the initialization python files.

[t-book]

Good point Florian, and thanks for your feedback. I'd say yes as all what should happen is that ENVs are overwritten. A look at docker-compose config gives more insights which environment is set for every servcei. However, this is definitely worth to check thoroughly

[vidlb]

What did you think about my remark on gitter, regarding .env which is used in a kind of misleading way in the default docker-compose.yml ?
Since it is a "global" exec env for compose, but isn't used by docker run by default...
As stated in the doc, .env should serve a substitution purpose and isn't automatically used for docker build or docker run. I read that .env should store mostly variables you intend to override using "$var" (with quotes) directly into your YAML files, OR override the override using host shell variables. https://github.com/docker/compose/issues/4223#issuecomment-280077263 It looks like even Docker compose doc example was using .env like this before and they changed it since.

Moreover putting all those vars in the same .env file is kind of brutal. Well I'm not complaining, it was a good reason to go deep in the advanced doc (again) to make sure I understood it well. As I've said on gitter, it's hard to understand which variable is required or not for each container... I do believe the best way is container-specific env_file directives in .yml, as in the core repo (db.env , django.env, geoserver.env), where you also store default values.
Then override using either shell, --env / --env-file, and/or the environment: section in the *override.yml file, since direct --env or ENV should take over .env or .env_prod files.

By the way, regarding --env and --env-file arguments, I just saw this is for Docker API v1.25+ . What is it about DOCKER_API_VERSION="1.24" I don't understand the issue mentioned just above this line in .env, but I guess it is related to ENV !

So right now, the script is peaking up global variables in this .env file, var which may get overriden with existing host shell variables.
Then it will also become container env via env_file: section in default YAML, when composition is up with docker run. But what is happening at this point to any variables which is defined in both .env and a second .env specified in .override.yml ? It is merging dicts, so the order of var definition would be ".env, .env_prod", thus one would expect that variables get overriden if they exists in the both files.

As we've seen with my weird problem (which was caused by the same kind of config that @t-book is suggesting here), I was able to read variables from the shell env (variables in .env.secret, which is called by docker-compose.override.yml), but django was unable to os.getenv() anything but default values... I thought this was about .env, I still do but there's something else. The only explanation I had was my entrypoint env beeing different than env in the shell (docker exec -it django4geonode /bin/bash). May be any variable that is in .env has precedence, I mean because it is in the default docker-compose.yml, could it be possible that any .env_* file which comes later with -f other_file.yml arg won't erase variables that were already declared ? This point is not really well explained in docker doc.

I'll try other configurations regarding this override thing, with a file which isn't named docker-compose.override.yml, and with different orders, so I'll be able to give you more insights !

[vidlb]

In the mean time regarding security, what about the override_prod_env.sh that I also suggested ? As @cwygoda_gitlab reminded on gitter, this does not address any security issues regarding the HOST.

But it would allow us to push "container commit" images : right know it would be a real bad idea to push a commit-container, because env is filled with secrets. But with a sh script override during entrypoint, we could make sure that our image is always storing only default passwords and keys, even after a commit.

There are many SO threads about this override thing. As I expected someone already suggested that a local ENTRYPOINT var namespace is a very good option.

Also there's https://github.com/docker/compose/pull/4368 and https://github.com/docker/docker.github.io/issues/9156 : in fact, secret files and variables may be used even if not in swarm mode. So this kind of overridesecrets.sh file (with variables exports) could store a secret (bind mounts in container /run during runtime only). Also it would allow to easily keep aligned with .env* files from upstream, if we're just editing new variables is this (untracked) sh file.

[cwygoda]

Hi, on github I'm just @cwygoda. :) Just a couple thoughts, since I'm not directly involved with geonode, but a professional user.

• override files for docker-compose are limited in nature (i.e. can't remove services)
• env files are somewhat hard to manage mentally, docker-compose config to the rescue!
• file base secrets would be a good investment, I'd guess a contribution from outside would be welcome
• in total, coming from a kubernetes+helm chart world, I started working with a middle layer for docker-compose based projects, where Jinja templates + vars+secrets targeted for a deployment environment render a final docker-compose.yml + other service configuration assets (i.e. for prometheus based metrics) which are then used locally or deployed remotely.

Explicit, separated configs (build, i.e. feature flags; runtime (whatever one uses, ie. local, devel, staging, canary, live) would make everyone's live easier.