bug report: CI failure for `contributors.yml` (_due to files with `CRLF` conflicting with `.gitattributes`_)

Description

The contributors.yml scheduled workflow recently failed which caught a mishap (my fault) with CRLF line endings committed in these files:

test/config/ldap/openldap/schemas/postfix-book.ldif
test/config/templates/dovecot-masters.cf
test/config/templates/postfix-accounts.cf

Insights:

Those files were introduced by me in https://github.com/docker-mailserver/docker-mailserver/pull/3514 before .gitattributes was added to prevent files checking in CRLF line endings.
The Github Action did have a successful run in-between those two PRs
I also noticed ECLint from the linting.yml workflow did not catch this mistake either (even though config implies it should have).
- UPDATE: This was because make lint actually uses this ecrc.json config instead which excludes linting the test/ directory.
- The test/ folder was excluded by: https://github.com/docker-mailserver/docker-mailserver/pull/2000
- Full history was shared for eclint config changes in my comment here. I raised the concern that it was not apparent why these excludes continued to be expanded (PRs introducing them did not provide much reasoning).

Logs

Contributors failing workflow log

https://github.com/docker-mailserver/docker-mailserver/actions/runs/6287887262/job/17072802215#step:6:53 ``` Create or update the pull request branch /usr/bin/git symbolic-ref HEAD --short master Working base is branch 'master' /usr/bin/git checkout --progress -B 7a7933b5-ccd5-4ecd-8447-70a6b72ee885 HEAD -- Switched to a new branch '7a7933b5-ccd5-4ecd-8447-70a6b72ee885' M test/config/ldap/openldap/schemas/postfix-book.ldif M test/config/templates/dovecot-masters.cf M test/config/templates/postfix-accounts.cf /usr/bin/git status --porcelain -unormal -- M test/config/ldap/openldap/schemas/postfix-book.ldif M test/config/templates/dovecot-masters.cf M test/config/templates/postfix-accounts.cf Uncommitted changes found. Adding a commit. /usr/bin/git add -A warning: in the working copy of 'test/config/ldap/openldap/schemas/postfix-book.ldif', CRLF will be replaced by LF the next time Git touches it warning: in the working copy of 'test/config/templates/dovecot-masters.cf', CRLF will be replaced by LF the next time Git touches it warning: in the working copy of 'test/config/templates/postfix-accounts.cf', CRLF will be replaced by LF the next time Git touches it /usr/bin/git -c author.name=github-actions[bot] -c author.email=41898282+github-actions[bot]@users.noreply.github.com -c committer.name=github-actions[bot] -c committer.email=41898282+github-actions[bot]@users.noreply.github.com commit -m docs: update `CONTRIBUTORS.md` --signoff [7a7933b5-ccd5-4ecd-8447-70a6b72ee885 3[57](https://github.com/docker-mailserver/docker-mailserver/actions/runs/6287887262/job/17072802215#step:6:63)8b7b] docs: update `CONTRIBUTORS.md` 3 files changed, 18 insertions(+), 18 deletions(-) ``` https://github.com/docker-mailserver/docker-mailserver/actions/runs/6287887262/job/17072802215#step:6:487 ``` From https://github.com/docker-mailserver/docker-mailserver * [new branch] contributors-update -> origin/contributors-update Pull request branch 'contributors-update' already exists as remote branch 'origin/contributors-update' /usr/bin/git checkout --progress contributors-update -- Switched to branch 'contributors-update' M test/config/ldap/openldap/schemas/postfix-book.ldif M test/config/templates/dovecot-masters.cf M test/config/templates/postfix-accounts.cf /usr/bin/git rev-list --right-only --count master...7a7933b5-ccd5-4ecd-8447-70a6b72ee885 1 /usr/bin/git rev-list --right-only --count master...contributors-update 0 /usr/bin/git diff --quiet contributors-update..7a7933b5-ccd5-4ecd-8447-70a6b72ee885 Resetting 'contributors-update' /usr/bin/git checkout --progress -B contributors-update 7a7933b5-ccd5-4ecd-8447-70a6b72ee885 -- error: Your local changes to the following files would be overwritten by checkout: test/config/ldap/openldap/schemas/postfix-book.ldif test/config/templates/dovecot-masters.cf test/config/templates/postfix-accounts.cf Please commit your changes or stash them before you switch branches. Aborting Error: The process '/usr/bin/git' failed with exit code 1 ```

Example of ECLint successfully catching this rule elsewhere

- Caught by a PR that introduced other files with CRLF by me in the `target/` folder: https://github.com/docker-mailserver/docker-mailserver/pull/3524#issuecomment-1704744596 - `.gitattributes` PR comment referencing it as justification for why the config was worth making explicit in git: https://github.com/docker-mailserver/docker-mailserver/pull/3527#issuecomment-1712225729 Workflow log: https://github.com/docker-mailserver/docker-mailserver/actions/runs/6070286374/job/16466057026#step:5:24 ``` Status: Downloaded newer image for mstruebing/editorconfig-checker:2.7.0 target/features/ldap/dovecot.base: Wrong line endings or no final newline Not all lines have the correct end of line character target/features/ldap/dovecot.tmpl: Wrong line endings or no final newline Not all lines have the correct end of line character target/features/ldap/postfix.base: Wrong line endings or no final newline Not all lines have the correct end of line character target/features/ldap/postfix.tmpl: Wrong line endings or no final newline Not all lines have the correct end of line character target/features/ldap/saslauthd.base: Wrong line endings or no final newline Not all lines have the correct end of line character target/features/ldap/saslauthd.tmpl: Wrong line endings or no final newline Not all lines have the correct end of line character 12 errors found [ ERROR ] ECLint failed make: *** [Makefile:76: eclint] Error 1 ```

Reproduction

$ git clone --depth=1 https://github.com/docker-mailserver/docker-mailserver .
$ git status

On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test/config/ldap/openldap/schemas/postfix-book.ldif
        modified:   test/config/templates/dovecot-masters.cf
        modified:   test/config/templates/postfix-accounts.cf

no changes added to commit (use "git add" and/or "git commit -a")

Understanding the problem cause

In a git repo, a file can have three representations:

Working Tree (the files on-disk from a checkout / clone)
Index (aka Staged changes via git add)
The repository (once committed via git commit)

These files were committed prior to the .gitattributes addition, thus git retained their line endings as-is (CRLF).

Once .gitattributes was added, git understood that these files are meant to have LF inside the repository state - but since the committed version is not LF, a diff is detected.

Where that can be a bit confusing initially, is that regardless of line ending being CRLF or LF in the working tree, this is still detected as a diff (to the repository copy with CRLF, that git knows needs to be committed as LF).
Once the file is staged into the index, if it was CRLF then the now staged (index) copy is converted to use LF (where the diff is against the repository version with CRLF).

Tip: Insights from `git ls-files --eol`

A good way to peek at the EOL usage within the Index and Working Tree by individual files (especially when troubleshooting / verifying .gitattributes rules) is to run git ls-files --eol.

We can combine that with grep to ignore all files recognized as LF in both the working tree and index:

# It may help to reset both working tree and index state to ensure this represents the repo copy:
$ git reset --hard HEAD

$ git ls-files --eol | grep -vE 'i/lf.*w/lf'

i/-text w/-text attr/-text              docs/content/assets/fonts/external-link.woff
i/-text w/-text attr/-text              docs/content/assets/logo/.src/dmo-logo-white.png
i/-text w/-text attr/-text              docs/content/assets/logo/.src/dmo-logo.png
i/none  w/none  attr/text               docs/content/assets/logo/dmo-logo-white.min.svg
i/none  w/none  attr/text               docs/content/assets/logo/dmo-logo.min.svg
i/-text w/-text attr/-text              docs/content/assets/logo/favicon-32x32.png
i/-text w/-text attr/-text              docs/overrides/assets/img/bg-water.webp
i/-text w/-text attr/-text              docs/overrides/favicon.ico
i/none  w/none  attr/text               target/postfix/header_checks.pcre
i/      w/      attr/text=auto          test/bats
i/crlf  w/crlf  attr/text               test/config/ldap/openldap/schemas/postfix-book.ldif
i/crlf  w/crlf  attr/text               test/config/templates/dovecot-masters.cf
i/mixed w/mixed attr/text               test/config/templates/postfix-accounts.cf
i/none  w/none  attr/text               test/config/without-accounts/.gitkeep
i/      w/      attr/text=auto          test/test_helper/bats-assert
i/      w/      attr/text=auto          test/test_helper/bats-support

-text is binary content, so that's all expected.
none means the file has a single line (potentially an empty file), EOL is missing (would add a blank line at end of file).
A blank value is for non-regular files, they are not in the index or accessible from the working-tree (in this case they are submodules for BATS).
mixed is usually not good. There is both CRLF and LF in that file, but thankfully with .gitattributes it was caught and being fixed (there are some files where mixed is valid, but it's rare).

NOTE: As I used git reset in the example above, both columns (i/ index + w/ working tree) are the same. However, on Windows when I checkout a branch:

i/ matches the repo committed EOL (normalized now by .gitattributes rules)
w/ checks-out text files with CRLF (when .gitattributes doesn't assign a specific eol, global git config defaults core.autocrlf=false + core.eol=native are used as fallback, hence CRLF on Windows), but only files that have a diff between the two branches/commits.
```
$ git ls-files --eol | grep -vE 'i/lf.*w/lf' | wc -l
247

# Compared to total files tracked by git which are checked out with LF:
$ git ls-files | wc -l
422
```
The amount depends on how many files changed between the branches.

EDIT: Not exactly, seems to be a specific kind of diff 🤷‍♂️ git reset would correct that (although I ran that via WSL, while the checkout was via a Windows GUI client).

TIP: VSCode (local config for Windows)

Prior to adding .gitattributes this resulted in some files that were LF being opened with CRLF in VSCode. I wasn't sure at the time what contributed to that.

With .gitattributes this is a non-issue as all text files can use LF (text attribute) when in the index and repo, and when necessary enforce the working-tree EOL with eol=lf / eol=crlf.
Using .gitattributes this way, makes the EOL in the working tree less of a concern, so the fallback core.eol=native can accommodate any local developer software that expects only LF or CRLF on that platform.
Although, it could be problematic in some scenarios like identified with a test failure.
- Test specific files with comparisons like this could likewise have their eol enforced in .gitattributes for consistency, or be generated 🤷‍♂️
- Alternatively, when core.eol=native does not make sense, nor core.eol=lf / core.eol=crlf; One can instead prefer core.autocrlf=input which will not perform any output conversion (overrides core.eol setting), so the working-tree checkout copy should match the committed EOL.
- git config --global core.autocrlf input will do this, but AFAIK changes the default for all git projects to normalize text EOL (crlf / mixed) to lf when staging changes to commit (which is probably what you'd want most of the time).
- Another alternative is leveraging our .editorconfig with an editor that supports it (such as VSCode with the official extension installed). This only applies the EOL conversion upon save however, thus not foolproof of working-tree gotchas.

TIP: Fixing the mismatch with `git add --renormalize .`

git add --renormalize . will automate resolving this.

Although I think staging the changes with just git add . may work in most cases? 🤷‍♂️ (I wanted to be sure, so discovered --renormalize from this SO answer, while this SO answer further explains it normalizes across all tracked files)

Running git rm -r --cached . && git reset --hard HEAD is needed afterwards (warning: Destructive to (un)staged changes).

This will correct the working tree (which when relying on CRLF conversion to LF by adding to the index, (instead of making the change prior via explicit conversion in the working tree) would still have the CRLF present in files) to reflect the EOL of now updated repository copy.
The same advice can be found from the Github Docs on handling line endings.

docker-mailserver / docker-mailserver