Quarkus dev Mode loading a Env file

[lbernardomaia]

Description Hi, Just an idea, would be nice to have the quarkus dev:mode accepting an env.file as an argument and all variables that eventually match one of the ${ENV_VARIABLE} in the application.properties, would be replaced.

Example: mvn quarkus:dev -Denv-file="/path/.env"

[dmlloyd]

So it would be a key-value file mapping environment variable names to values that would be added to the environment of the dev process?

[machi1990]

@dmlloyd @lbernardomaia I can take a look at it.

So it would be a key-value file mapping environment variable names to values that would be added to the environment of the dev process?

Std .env file I suppose?

[lbernardomaia]

@dmlloyd, correct!

Another example:

.env

RED_APPLE_ENDPOINT=http://redapple.dev.com
GREEN_APPLE_ENDPOINT=http://greenapple.dev.com
BANANA_ENDPOINT=http://banana.dev.com

application.properties

redapple.endpoint=${RED_APPLE_ENDPOINT}
greenapple.endpoint=${GREEN_APPLE_ENDPOINT}
banana.endpoint=${BANANA_ENDPOINT}
grape.endpoint=${BANANA_ENDPOINT}

If I run mvn quarkus:dev -Denv-file="/path/.env", the keys redapple.endpoint, greenapple.endpoint and banana.endpoint would be replace for the correspondent values in the .env file.

[stale[bot]]

This issue/pullrequest has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gsmet]

PR still open.

[gsmet]

@lbernardomaia could you explain why you can't use the %dev. profile? It seems to fit your requirements?

Or do you want to be able to override things on a per user basis? Or you even want to have several local environments?

I think we really need to understand what you're trying to do before coming with a solution.

[mejlholm]

I also think this would be useful. Typically we create a secrets.env file in each project with local setup. This file is used to create a Kubernetes secret when testing image locally. To re-use the secrets.env file we “source” the it before starting quarkus:dev. It would be nice to be able to refer to the secrets.env file instead of contaminating the shell with the contents of the file.

[lbernardomaia]

The point made by @mejlholm it's also my point. If I have an application running in a container and the only thing that my application is aware of is environment variables, I need to get them somehow. One way to do that and keep consistency is to use the application.properties to map properties to env variables as described previously.

redapple.endpoint=${RED_APPLE_ENDPOINT}

When I run quarkus on Dev mode, I have to set those environment variables on my local environment, otherwise, my app doesn't work, and I have to do that for each application. With the proposal, I would have the flexibility to have a .env file per application, specify that to Quarkus quarkus:dev -Denv-file="/path and it would replace possible matches on application.properties.

Note that I only have one application.properties per application, when my container is deployed on other environments, the only thing that changes is environment variables.

[dmlloyd]

The point made by @mejlholm it's also my point. If I have an application running in a container and the only thing that my application is aware of is environment variables, I need to get them somehow. One way to do that and keep consistency is to use the application.properties to map properties to env variables as described previously.

redapple.endpoint=${RED_APPLE_ENDPOINT}

You should already be able to supply an env var named REDAPPLE_ENDPOINT (mapping to uppercase and replacing . with _) and thereby override this value without having to specify it in the configuration file at all. In 1.0 there are some restrictions to when this can happen, but in the upcoming 1.1 it should already work very uniformly.

When I run quarkus on Dev mode, I have to set those environment variables on my local environment, otherwise, my app doesn't work, and I have to do that for each application. With the proposal, I would have the flexibility to have a .env file per application, specify that to Quarkus quarkus:dev -Denv-file="/path and it would replace possible matches on application.properties.

FWIW any application.properties file found on the filesystem at run time will be used at a higher priority than the one that the app was built with. That might help in scenarios like this.

Note that I only have one application.properties per application, when my container is deployed on other environments, the only thing that changes is environment variables.

This is a different use case altogether, but fortunately this is one that should already work today (see above).

[dmlloyd]

I also think this would be useful. Typically we create a secrets.env file in each project with local setup. This file is used to create a Kubernetes secret when testing image locally. To re-use the secrets.env file we “source” the it before starting quarkus:dev. It would be nice to be able to refer to the secrets.env file instead of contaminating the shell with the contents of the file.

I really feel that secrets should not directly be part of the configuration. All it takes is one helpful tool to log all of the configuration and your private keys are on the console.

We need a separate indirection mechanism for this, i.e. a broad, formal vault/credential store mechanism.

[dmlloyd]

Here is a clear, well-defined use case contributed by @sherl0cks, who brought this idea up in chat (please let me know if I have an error Justin!).

Use case: As a developer, I want to be able to specify an .env file in the working directory of my project build or execution, so that I can add "environment" variables which apply only to my local build/execution in a way which is already well-understood and idiomatic for other microservice frameworks.

Notes: Java doesn't support actual environment modification. @sherl0cks mentioned java-dotenv, a project which "cheats" a bit by providing a separate environment API (as this is the only practical portable approach to this problem in Java), but in our case it would probably make more sense to co-opt io.quarkus.runtime.configuration.ConfigUtils.EnvConfigSource to do this since we're already funneling everything through SmallRye Config anyway. As a standardization note, it might be worth proposing that .env be supported within the MP Configuration specification generally in this manner.

While we could have easily turned this into a request to support a "local" version of application.properties, the particulars of this issue revolve around the ability for users who have familiarity with existing microservice-related frameworks who might (reasonably) expect to use the same kind of mechanism. So treating the file as if it were a list of environment variable names and values - in particular, with the corresponding syntax/configuration implications - is significant in terms of use case, which is why I've spelled it out explicitly.

If we do want to generalize a "local" application.properties, a separate issue should be opened for that purpose.

[sherl0cks]

@dmlloyd you have the right Justin. Thanks for tagging me here. As examples of how standard this is in other languages, all of the below examples have at least 2,000 github stars:

• Node.js
• Ruby
• Python
• Go

If it's helpful, here is the gist of the config source I wrote for this (in Kotlin).

[geoand]

What is the standard location that we expect users to place this file in? Asking because of #7519

[dmlloyd]

There is no standard location because it is read from the current working directory, so wherever you are is where it goes.

[geoand]

Hm... I am really wondering in other runtimes that support the feature, where are users expected to place the file so they have use it during the development cycle? The root of the project seems to make sense to me, but I have no experience with it...

[geoand]

Reading this I think it's expected to be in the root directory and not be added to source control

[machi1990]

Reading this I think it's expected to be in the root directory and not be added to source control

Yes, they are not meant to be committed and especially not meant to be used in production environment.

[mejlholm]

+1 for root folder.

Don't forget to add the file to a .gitignore file as well.

[geoand]

+1 for root folder.

Don't forget to add the file to a .gitignore file as well.

That would probably be up to the developer to do since we won't generate a .env file by default

[mejlholm]

That would probably be up to the developer to do since we won't generate a .env file by default

I'll check once its done and make sure it's in /gets into the guide :-)

[lbernardomaia]

Hi all,

I was testing this change and I noticed that when an env variable is not provided in the .env file, quarkus it's not falling back to try to use the env system. Is this expected? Thanks

[geoand]

@lbernardomaia

You mean that if you do have a .env file but that doesn't contain an env var (like QUARKUS_HTTP_PORT), then if you do set that (using export for example), it's not applied?

[geoand]

I was not able to reproduce what you mention @lbernardomaia, but I might have understood it wrong.

[lbernardomaia]

Hi @geoand ,

My bad! You're right, It's working with all scenarios I tried.

Thanks

[geoand]

Thanks for checking @lbernardomaia!