Dotenv Implementation Comparison Framework

[casey]

Yooooo!

I wrote a framework for comparing dotenv implementations: https://github.com/casey/dotenv-compare

Right now it only compares a few implementations, doesn't have a lot of test cases, and it doesn't deal gracefully with implementations crashing, so it's more of a proof-of-concept than anything else.

At the very least, I'd like to (hopefully with help!) add more test cases to it, and use it to make sure that dotenv-rs is in conformance with the ruby implementation. (Assuming that is a goal for dotenv-rs!)

Once it has a comprehensive set of test cases that exercise all aspects of popular implementations, I think it would be useful to think about creating a standard for .env files, hopefully with support from other implementations. Ideally, new implementations could be written with the aid of a formal grammar, and tested against a set of test cases, instead of using the Ruby implementation as an ad-hoc standard.

Do you think this is something that could live in the dotenv-rs organization?

[ZoeyR]

I would love to see the dotenv file format become standardized across all the various implementations. The dotenv-rs organization was meant originally for just managing the rust implementation, but I wouldn't be opposed to changing this to manage the spec as well. Just as long as we can get the other dotenv projects on-board.

[casey]

Do you know how other dotenv projects are thinking about compatibility? Are they trying to be compatible with the ruby implementation, or is that a non-goal? Is the ruby implementation the original and still most popular implementation?

I'm not sure what the best way to tackle the project would be, but probably just mapping the space of implementations and their features and compatibility would be a good start.

[ZoeyR]

It is my understanding that the Ruby implementation is the original. Most implementations are probably trying to emulate the ruby one as closely as possible (I know this project is), but I can't say for sure. I think it would be best to also bring in the maintainer of the ruby gem into this discussion so that we can decide if standardizing on a unified grammar is a goal that multiple parties would be interested in.

[casey]

That sounds reasonable.

Pinging @bkeepers and @cadwallion!

For background, @ZoeyR and I have been discussing ways that the different dotenv implementations might be brought into parity. (Which probably means that they all do what the Ruby one does, since that's what users expect.)

A little bit of background on why I'm interested in helping implementations reach parity: I wrote a command runner thing called Just, which supports loading .env files. It's in Rust, so it uses dotenv-rs. But I think users expect it to be compatible with whatever dotenv implementation they're used to, so sometimes dotenv-rs's behavior is surprising to them.

I wrote a comparison framework for comparing dotenv implementations, but I'm actually starting to think that isn't a very good approach. A much simpler way to go about it would be to factor the Ruby implementation's test cases into a common format, like YAML, JSON, or CSV. That way, the non-ruby implementations could use all of the Ruby implementation's test cases, and could thus ensure that they have the same behavior.

Does that sound promising?

[bkeepers]

I think unified test cases are a great idea. It's something I wish I had done from the beginning. I haven't been actively maintaining dotenv lately, so I defer to @cadwallion on plans for the Ruby implementation.

One bit of advice: I would suggest writing the test cases against bash as well, and treat it as the authoritative implementation. You could even write the shared test suite in bash, and call out to the language specific executable that prints the state of the environment after evaluating a .env file and compares the result against bash.

Something like this:

#!/bin/bash

EXPECTED="$(source test/case1.env && env)"
ACTUAL="$(bundle exec dotenv -f test/case1.env print-env.rb)"
test '"$EXPECTED" = "$ACTUAL"'

[casey]

I think treating bash as authoritative is a great idea. There are a bunch of different .env implementations that differ wildly in what they support, but they all (with a few exceptions) more or less support bash-compatible syntax.

[mathroc]

if you want to test with bash, you've got to use the allexport option, and probably clearing the env:

EXPECTED=$(env -i bash -c 'set -a; source test/case1.env; set +a; env)