Add configuration provider for environment (.env) files

[karolz-ms]

Summary

I propose that we add a configuration provider for .env files.

They are very popular and convenient in the Linux/Docker/Docker Compose world. It would significantly simplify development environment configuration for services that use ASP.NET Core and require dependencies (supporting services) that run as containers. These application- and supporting services often require shared pieces of configuration, and keeping them in a single .env file is very convenient.

Describe the solution you'd like

A Microsoft-authored file configuration provider, similar to existing file configuration providers like the INI- or JSON ones.

I have a prototype:

class EnvFileConfigurationProvider : FileConfigurationProvider
    {
        public EnvFileConfigurationProvider(EnvFileConfigurationSource source) : base(source) { }

        public override void Load(Stream stream)
        {
            var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
            var doubleQuotedValueRegex = new Regex(@"""([^""\\]*(?:\\.[^""\\]*)*)""");
            var singleQuotedValueRegex = new Regex(@"'([^'\\]*(?:\\.[^'\\]*)*)'");

            using (var reader = new StreamReader(stream))
            {
                while (reader.Peek() != -1)
                {
                    string line = reader.ReadLine().Trim();
                    if (string.IsNullOrWhiteSpace(line))
                    {
                        continue;
                    }

                    if (line.StartsWith("#", StringComparison.Ordinal))
                    {
                        continue; // It is a comment
                    }

                    int separator = line.IndexOf('=', StringComparison.Ordinal);
                    if (separator <= 0 || separator == (line.Length-1))
                    {
                        continue; // Multi-line values are not supported by this implementation.
                    }

                    string key = line.Substring(0, separator).Trim();
                    if (string.IsNullOrWhiteSpace(key))
                    {
                        throw new FormatException("Configuration setting name should not be empty");
                    }

                    string value = line.Substring(separator + 1).Trim();

                    var doubleQuotedValue = doubleQuotedValueRegex.Match(value);
                    if (doubleQuotedValue.Success)
                    {
                        value = doubleQuotedValue.Groups[1].Value;
                    }
                    else
                    {
                        var singleQuotedValue = singleQuotedValueRegex.Match(value);
                        if (singleQuotedValue.Success)
                        {
                            value = singleQuotedValue.Groups[1].Value;
                        }
                        else
                        {
                            int commentStart = value.IndexOf(" #", StringComparison.Ordinal);
                            if (commentStart > 0)
                            {
                                value = value.Substring(0, commentStart);
                            }
                        }
                    }

                    data[key] = value;
                }
            }

            Data = data;
        }
    }

Additional context

In https://github.com/hashicorp/terraform/issues/23906 a user is asking to add support for .env files to Terraform and they also list libraries that allow .env files to be consumed from Node, Python, Ruby, and Go. The Node package has 15M weekly downloads and 13k GH stars.

There is a community NuGet package to parse .env files, but I propose we go a step further and have a Microsoft-authored configuration provider for these files added. It would make it much simpler to take dependency on this configuration provider from project templates for VS/VS Code, and incorporate it in examples.

[karolz-ms]

@davidfowl @glennc FYI

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @maryamariyan, @safern See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Summary I propose that we add a configuration provider for .env files. They are very popular and convenient in the [Linux/Docker/Docker Compose world](https://docs.docker.com/compose/env-file/). It would significantly simplify development environment configuration for services that use ASP.NET Core and require dependencies (supporting services) that run as containers. These application- and supporting services often require shared pieces of configuration, and keeping them in a single .env file is very convenient. ### Describe the solution you'd like A Microsoft-authored file configuration provider, similar to [existing](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-5.0#file-configuration-provider) file configuration providers like the INI- or JSON ones. I have a prototype: ```csharp class EnvFileConfigurationProvider : FileConfigurationProvider { public EnvFileConfigurationProvider(EnvFileConfigurationSource source) : base(source) { } public override void Load(Stream stream) { var data = new Dictionary(StringComparer.OrdinalIgnoreCase); var doubleQuotedValueRegex = new Regex(@"""([^""\\]*(?:\\.[^""\\]*)*)"""); var singleQuotedValueRegex = new Regex(@"'([^'\\]*(?:\\.[^'\\]*)*)'"); using (var reader = new StreamReader(stream)) { while (reader.Peek() != -1) { string line = reader.ReadLine().Trim(); if (string.IsNullOrWhiteSpace(line)) { continue; } if (line.StartsWith("#", StringComparison.Ordinal)) { continue; // It is a comment } int separator = line.IndexOf('=', StringComparison.Ordinal); if (separator <= 0 || separator == (line.Length-1)) { continue; // Multi-line values are not supported by this implementation. } string key = line.Substring(0, separator).Trim(); if (string.IsNullOrWhiteSpace(key)) { throw new FormatException("Configuration setting name should not be empty"); } string value = line.Substring(separator + 1).Trim(); var doubleQuotedValue = doubleQuotedValueRegex.Match(value); if (doubleQuotedValue.Success) { value = doubleQuotedValue.Groups[1].Value; } else { var singleQuotedValue = singleQuotedValueRegex.Match(value); if (singleQuotedValue.Success) { value = singleQuotedValue.Groups[1].Value; } else { int commentStart = value.IndexOf(" #", StringComparison.Ordinal); if (commentStart > 0) { value = value.Substring(0, commentStart); } } } data[key] = value; } } Data = data; } } ``` ### Additional context In https://github.com/hashicorp/terraform/issues/23906 a user is asking to add support for .env files to Terraform and they also list libraries that allow .env files to be consumed from Node, Python, Ruby, and Go. The Node package has 15M weekly downloads and 13k GH stars. There is a [community NuGet package](https://www.nuget.org/packages/dotenv.net/) to parse .env files, but I propose we go a step further and have a Microsoft-authored configuration provider for these files added. It would make it much simpler to take dependency on this configuration provider from project templates for VS/VS Code, and incorporate it in examples.

Author:	karolz-ms
Assignees:	-
Labels:	`area-Extensions-Configuration`, `untriaged`
Milestone:	-

[jongio]

Suggestions:

1. Include env var expansion. If the file contains a reference to an env var, then auto-expand it or provide an option to do so if not done by default

i.e.

  export BASENAME=foo

  BASENAME=${BASENAME}

will auto expand to:

  BASENAME=foo

2. Implement a generic .env file parser so it can be used stand-alone.

[spboyer]

@davidfowl @glennc - +1

Would compliment Azure Developer CLI patterns and meet other language stack support for .env files.

[davidfowl]

@spboyer How do you imagine this would be used in practice?

[spboyer]

@davidfowl the tldr; is much like it is used for JS or Python. It is a good compliment to what dotnet is already doing with INI, JSON, etc in Microsoft.Extensions.Configuration. A community OSS project DotEnv is a start.

Such as API keys and database credentials, separate from the codebase. JavaScript and Python developers use the .env file in a similar way. I would suggest primarily for local dev and shared .env for polyglot devs usind .env files.

Here are a few examples:

Setting up environment variables: Developers use the .env file to store environment variables that are specific to the project, such as the database connection URL, API keys, and authentication tokens.

Keeping sensitive information secure: Sensitive information such as passwords, keys, and other secrets are kept in the .env file, which is not committed to version control. This ensures that the information is kept safe and secure.

Configuring application settings: Developers can use the .env file to store application settings that can be easily configured, such as the port number, the log level, and other application-specific settings.

To use the .env file in JavaScript or Python, developers typically use a package such as dotenv or python-dotenv to load the environment variables from the file into the application. Once loaded, the variables can be accessed like any other environment variable within the code.

[jongio]

@davidfowl - Here's where we discussed this: https://youtu.be/YkEl16UsuHA?t=367

[davidfowl]

I'm looking for what the application would look like with this feature? Does the app check in code that references a file that isn't in the repository? Is that what people typically do?

[spboyer]

They wouldn't check in the .env files, similar to how .net uses secrets. In Python you'd load the "dot" files and refer to the ENV and either their loaded from the .env or ENV

# Load environment variables from .env file
load_dotenv()

# Access environment variables
db_host = os.getenv('DB_HOST')
db_user = os.getenv('DB_USER')
db_pass = os.getenv('DB_PASS')

[davidfowl]

They wouldn't check in the .env files, similar to how .net uses secrets.

User secrets are in a well known path. What do you suggest for env files?

[spboyer]

Typically the .env files would be at the root, by practice this is easiest to exclude from source but obviously could be in any subdirectory, but then you need to specify the path in dotenv similar to dotnet with INI etc.

dotenv_path = os.path.join(os.getcwd(), 'config', '.env')
load_dotenv(dotenv_path)

[davidfowl]

So the issue is asking for a configuration provider. This is your typical ASP.NET Core application:

var builder = WebApplication.CreateBuilder(args);

// builder.Configuration.AddEnvFile( what path goes here?)

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

What would the configuration provider wire up look like in this case?

[spboyer]

var envFilePath = @"C:\path\to\file.env"
builder.Configuration.AddEnvFile(envFilePath)

I think that it is simply modeled after how we handle .AddIniFile initially.

[davidfowl]

So in practice, you commit code that points to files outside of the repository. I'm not a fan of this approach. What happens if I move across platforms? Should the path be relative?

I see the dotenv project either does a directory search for the env file or allows you to optionally copy the env file to the output (which would make it more portable), but maybe less useful? I don't know. I know env files are commonly used in other stacks (including dock

[spboyer]

Sorry perhaps poor file location sample path. Here is better what I was trying to covey.

var builder = new ConfigurationBuilder()
    .SetBasePath(AppContext.BaseDirectory)
    .AddEnvFile("file.env", optional: true, reloadOnChange: true);

this being at the root of the project, as mentioned being normal practice for other languages. We would promote reaching outside of the repo.

[davidfowl]

This dotenv project looks so much more complete than what's being suggested here, and works in general for all environment variables. Do you still think this is valuable to add as a config provider?

[spboyer]

.NET has great support for Environment variables already, but is missing .env support. I believe that "in the box" to extend known file type/pattern across stacks is a good choice.

[davidfowl]

That doesn't answer the question though. IMO the dotenv OSS project does a better job of adding .env support to .NET than what is being suggested here.

To be specific:

• This would be a configuration provider, it doesn't match what dotenv generally (setting environment variables)
• I'm not sure how the .env file is discovered. It seems like there needs to be a convention around how these files are discovered that doesn't lead users into a pattern where they end up checking in physical paths.

[karolz-ms]

The dotenv OSS project is not integrated with the .NET configuration system, that is the main drawback of this OSS package and the core value prop of what this proposal is about.

The pattern that would be great to enable is that .env is an optional configuration source, overriding/augmenting other sources as necessary. By "optional" I mean if the file is not found, well then nothing is contributed, but execution continues. The file itself is located in a program-specific, but well-known location using a relative path. This would be very useful for development scenarios.

[davidfowl]

The dotenv OSS project is not integrated with the .NET configuration system, that is the main drawback of this OSS package and the core value prop of what this proposal is about.

Understood, but it does is much more complete than the above. Adding this provider to .NET would be fine as it wouldn't have any dependencies, but it's missing some important details.

By "optional" I mean if the file is not found, well then nothing is contributed, but execution continues. The file itself is located in a program-specific, but well-known location using a relative path. This would be very useful for development scenarios.

Thats how most of the configuration providers work, but I'm asking some very specific questions that I think need to be answered if we are to move this forward.

[spboyer]

While dotenv.net may be a more comprehensive solution, it can also add complexity to a project by introducing another third-party dependency. By providing a native configuration provider for .env files, .NET can offer a simple, built-in solution that doesn't require additional dependencies.

What other questions do you need to have answered to move this forward beyond why we support file formats like INI, JSON, XML in a similar way?

[davidfowl]

Are you thinking that these files would be in source control then? Like the JSON and ini files are checked in..

[jongio]

.env files aren't committed. Usually people will commit a .env.template and ask people to rename and populate it. That way they have the expected key names.

[davidfowl]

I know, that's why I'm challenging the end-to-end experience here. We're saying it's just like JSON and XML and on the other hand, it's different in that you don't use them in a similar way. Or maybe you're just saying that we would show them in the project folder but always in the gitignore ....

[bugproof]

.env files are very common in non .NET ecosystems (you got .env and .env.local) for example in next.js. I also think they're much easier during development than user secrets. Yeah they can leak but you will most likely use only sandbox/development keys with them. I find it mind-boggling that official parser /configuration provider is not available for it