CLI Support for Injecting Envars into Docker-Compose Containers without embedding the CLI in Container Images

dangtony98 commented 1 year ago

Feature description

Currently, the Infisical CLI doesn't have a good way of injecting environment variables into services with docker-compose without needing to embed it into the image of each service.

The proposed method is to have the CLI inject environment variables into each docker container by wrapping the docker-compose up command.

Why would it be useful?

It would likely be incredibly impactful for anyone using docker-compose since they won't even have to embed the CLI into their docker images anymore.

Additional context

It would be cool to define a mapping from project ids to docker services for which service should be injected the environment variables from which Infisical project; then upon starting up with infisical run -- docker-compose up, it should inject according to that mapping — This has some parallels to the existing branching mechanism.

caioluis commented 1 year ago

I agree. To be honest, I don't like the idea to feed the Docker image with a token at all, although I haven't put much thought into the security implications, so I don't know if it is in fact a problem. By common sense, I would say that embedding the token in the image is not good. And this is what we are doing now.

May someone enlighten me? I may be under a misconception.

caioluis commented 1 year ago

Although I'm not knowledgable enough about the codebase, I will just spit some initial ideas as I run through the code:

It might be possible to use the same logic applied to executeSingleCommandWithEnvs and executeMultipleCommandWithEnvs when working with Docker. If we detect that what follows the run command is a docker like engine command (docker or nerdctl, idk), we can further pass it to Docker following the instructions from Docker's documentation.
- From what I understood from the Infisical code base, on executeSingleCommandWithEnvs we parse the actual command with command := args[0]. We could use that to identify the docker command. I modified the cmd package to test this hypothesis:
- Then, I guess it could be a matter of reordering the injection of the secrets. Right now, we store the env list on cmd.Env. Instead of storing it this way, we could append the envs to the fullComand.

I'm not used to Go and the codebase, but I put some effort the past day to understand how things are going. I'm quite unexperienced and I don't know if what I'm saying makes sense.

One of my concerns is the other env vars we embed in the shell command. I noticed that the env injects around 40 other env vars that are probably local settings of my own dev environment. I honestly don't know how it works.

That's it, I would love to hear your opinion.

maidul98 commented 1 year ago

One of my concerns is the other env vars we embed in the shell command. I noticed that the env injects around 40 other env vars that are probably local settings of my own dev environment. I honestly don't know how it works.

This would happen even if you ran a command without infisical. What Infisical does is to pass on all the environment that would already be exposed and just adds Infisical secrets on top

For your apprach are you thinking of using this method from the docs? https://docs.docker.com/compose/environment-variables/set-environment-variables/#use-the-environment-attribute

I think this method will work without any changes since docker compose already reads from the environment variables. The issue is you will have to write out all the keys from Infisical secrets manually. For this, I am thinking that we need to use some kind of templating lanaguge to write out all the keys in the compose file before starting compose up

caioluis commented 1 year ago

This would happen even if you ran a command without infisical. What Infisical does is to pass on all the environment that would already be exposed and just adds Infisical secrets on top

Ok! Now I see, thanks for the info

For your apprach are you thinking of using this method from the docs? https://docs.docker.com/compose/environment-variables/set-environment-variables/#use-the-environment-attribute

Not, I was thinking in running as options (-e, --env). https://docs.docker.com/compose/environment-variables/set-environment-variables/#set-environment-variables-with-docker-compose-run---env

I think this method will work without any changes since docker compose already reads from the environment variables. The issue is you will have to write out all the keys from Infisical secrets manually. For this, I am thinking that we need to use some kind of templating lanaguge to write out all the keys in the compose file before starting compose up

Hmm, I don't think writing to the compose file is necessary. I'm still playing around with this implementation, but I managed to run build a simple container image that checks the existence of one env var, and the method that I used created a working container with that env in it.

Of course, the problem comes when we need to update the env vars. On my research, I saw many people complaining that updating containers' env vars is not that straightforward. container service update, for instance, can only be used with Swarm; container start does not support -e, --env as options, intentionally.

However, this particular answer on SO shows a way to create a new container from the same image, yet preserving the volume from the old container.

What do you think? I'll be opening a WIP PR so we can discuss it in the actual code.

maidul98 commented 1 year ago

hey @caioluis, I wanted to follow up on our Slack conversation. I believe it's worth exploring other options for adding environment variables to docker-compose, since the current approach isn't compatible with running docker-compose up and requires individual service startup

philmas commented 9 months ago

This would be really beneficial. I currently struggle with docker compose because infisical can only inject inside the container, but not outside. So if you have:

container_name: xyz
image: abc
ports:
      - 127.0.0.1:${PORT}:${PORT}
environment:
      INFISICAL_TOKEN: ${INFISICAL_TOKEN}
      INFISICAL_API_URL: ${INFISICAL_API_URL}

${PORT} will not be injected here.

Infisical / infisical