search

ldc-developers / ldc

The LLVM-based D Compiler.
http://wiki.dlang.org/LDC
Other
1.21k stars 261 forks source link

CRLF terminators in release tarball files #2402

Open ximion opened 6 years ago

ximion commented 6 years ago

Hi! This has been messing with the Debian packaging for a while now, which is why I repackaged the release tarball every single time when adding LDC to Debian, removing CRLF from files. Adding those files to Git will result in Git removing CRLF automatically, resulting in a diff between the original tarball and our packaging sources in Git, which causes problems.

Since Git doesn't accept these files, I wonder how they appear at all in the release tarballs, and it would be neat to get this resolved in future.

Here is a list of the affected files:

./tests/d2/dmd-testsuite/compilable/ldc_github_982.d: C source, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/ldc_github_421.d: C source, UTF-8 Unicode (with BOM) text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/ldc_github_1211.d: C source, UTF-8 Unicode (with BOM) text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc11479.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc7656.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc8271.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc13645.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc8.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10325.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc6491.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc2.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddocunittest.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc13.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc5446.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9676a.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc7.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc12706.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc7715.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc15475.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddocbackticks.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc4.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9789.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc11511.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10869.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9369.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc6.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc14778.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc11823.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc8739.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10367.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10366.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9497c.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc3.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc4162.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc198.html: HTML document, ASCII text, with very long lines, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc12.html: HTML document, UTF-8 Unicode text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc1.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc14383.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc5.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc11.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc14.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9497b.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10870.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc7795.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9037.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9497d.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9155.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9727.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9764.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc17169.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc12745.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9497a.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc10334.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9676b.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9903.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc648.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9475.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddocYear.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc13270.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc9305.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc7555.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/compilable/extra-files/ddoc2273.html: HTML document, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/runnable/ldc_github_328.d: C source, ASCII text, with CRLF line terminators
./tests/d2/dmd-testsuite/runnable/ldc_github_1058.d: C source, UTF-8 Unicode (with BOM) text, with CRLF line terminators
./tests/d2/dmd-testsuite/runnable/ldc_github_989.d: C source, UTF-8 Unicode (with BOM) text, with CRLF line terminators
./tests/d2/dmd-testsuite/runnable/helloUTF16BE.d: C source, Big-endian UTF-16 Unicode text, with CRLF line terminators
./tests/d2/dmd-testsuite/runnable/helloUTF16.d: C source, Little-endian UTF-16 Unicode text, with CRLF, CR line terminators
./tests/d2/dmd-testsuite/fail_compilation/testhtml2.html: HTML document, ASCII text, with CRLF, CR, LF line terminators
./runtime/druntime/src/ldc/msvc.c: C source, ASCII text, with CRLF, LF line terminators
./runtime/druntime/src/ldc/eh/msvc.d: C source, ASCII text, with CRLF, LF line terminators

Thanks! Matthias

kinke commented 6 years ago

Adding those files to Git will result in Git removing CRLF automatically

Depends on the git core.autocrlf config. Most of these dmd-testsuite files have these Windows or even mixed line endings because they are like that in the DMD git repo (and we have a script automating the update of our derived dmd-testsuite git submodule). The few LDC specific tests are a different matter of course.

ximion commented 6 years ago

So, should I report this to the DMD project directly?

kinke commented 6 years ago

I guess it won't hurt; I'm not sure whether we can fix it on our side without breaking the automation script. :/

dnadlinger commented 6 years ago

Is core.autocrlf enabled on the boxes used to prepare the Debian releases or somewhere in the automatic build chain? I'd discourage using it if you have a sensible editor (i.e. not Notepad on Windows), as it has always been more trouble than it is worth for me. Just leaving the sources as CRLF should work fine.

ximion commented 6 years ago

No, there is no core.autocrlf anywhere (I didn't even know about it). The problem is that Git transparently removes CRLF, so the Git repo contains technically different sources than the release tarball. And that messes with git-buildpackage and dpkg when building packages.

Since the amount of files using CRLF is so incredibly small, I would guess fixing this once is much easier than (continuing to) apply workarounds.

ximion commented 6 years ago

FTR, this is reported as https://issues.dlang.org/show_bug.cgi?id=18027 to DMD now.

dnadlinger commented 6 years ago

The problem is that Git transparently removes CRLF, so the Git repo contains technically different sources than the release tarball.

Which "Git" and where does the removing? To my knowledge, Git doesn't "transparently" mess with line endings, unless instructed to do so. core.autocrlf is the setting that controls that. I don't think it is enabled by default.

the Git repo contains technically different sources than the release tarball

Which release tarball are you talking about? From your original post I got the impression that our source archives on the GitHub release page contain the mixed endings (correct), but they are normalized to LF somewhere on the Debian side (incorrect/problematic). But perhaps this was a misunderstanding?

ximion commented 6 years ago

FWIW, Debian's tools apparently enforce this setting: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719363 This is highly annoying.