dotnet new creates files with CRLF line endings on linux

[livarcocc]

From @kromatic on September 21, 2017 20:58

Steps to reproduce

Run dotnet new classlib in new project folder.

Expected behavior

On Linux, this should create Class1.cs and .csproj files with LF line endings.

Actual behavior

Class1.cs and the .csproj file are both created with CRLF line endings. (Files in bin and obj folders appear to all have the correct LF line endings.)

Environment data

dotnet --info output:

.NET Command Line Tools (2.0.0)

Product Information: Version: 2.0.0 Commit SHA-1 hash: cdcd1928c9

Runtime Environment: OS Name: fedora OS Version: 26 OS Platform: Linux RID: fedora.26-x64 Base Path: /usr/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

Version : 2.0.0 Build : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

Copied from original issue: dotnet/cli#7695

[mlorbetske]

This is currently "by design" (see https://github.com/dotnet/templating/issues/350#issuecomment-281851892). Originally, we had standardized all content on LF line endings, but this caused issues with certain editors on Windows and produced a ton of messages when adding the files to Git.

Since the content is not produced line by line, we don't explicitly set what the line endings should be, we just emit what was included in the original content. We are looking in to a feature in the future (if there's enough interest) where templates could indicate that files matched by certain patterns would have their line endings replaced with ones matching the platform.

[SteffenL]

I just ran dotnet new [...] for the first time on Linux, and I was rather shocked to see CRLF line endings as I staged the files in the Git repository.

I would personally want native/expected behavior from a cross-platform tool, regardless of what elementary things certain editors have problems with. For example, if an editor (let's say Notepad on Windows) can't deal with LF, then I would rather fix the editor or use a different one.

At least for now, it seems like git config --global core.autocrlf input will work okay for me personally on Linux and macOS, and git config --global core.autocrlf true on Windows.

[sandyarmstrong]

This is incredibly frustrating on non-Windows platforms.

If you forget about this bug before you start making changes to your code, you can easily end up with mixed line endings, which confuses all sorts of tooling.

Fixing it for the whole repo requires dos2unix and, realistically, more shell scripting know-how than most devs possess to avoid accidentally modifying the contents of .git and node_modules, etc.

[laktak]

Could we get an update on the current status of this bug?

The last comment in the referenced issue says:

This is now fixed

but dotnet new still produces CRLF?

[michelesolytic]

I also encountered this problem today, and I disapprove the decision of keeping this behaviour by design. If there are some editors on Windows which have troubles handling LF endings, then it should be their responsibility to fix their behaviour, not yours. I believe you should rather provide instructions to those users (or a command flag, or a tool) to counter the issue in those scenarios: once the users are aware of the problem, they'd eventually push the editor's developers to fix it.

[laymanmu]

Carriage Returns were designed for mechanical typewriters. What year is it again? Is there seriously no option to tell the dotnet cli to not add carriage returns? There's no config file or environment variable? What bubble do you live in to think that this is Ok? I think the emperor needs to put on some clothes - I'm here to tell you that the world does not use CRLF anymore and if you depend on it then you're being left behind.

[volkoff-pro]

This behavior of dotnet cli is very frustrating for me. I'm waiting maybe some flag for line ending, or something else like this. If I generating big project using built-in templates it is very hard to change crlf to lf(

[genalt]

Looks like this issue is resolved. I have just checked on Linux, all files contain LF endings. The issue will be closed next triage session.

[n-romaniv]

@grinrag, hi! It still repduces for me. It is very frustrating because OmniSharp in VSCode seems to work only when the files have LF as line endings. Otherwise, there is no highlighting or hints.

.NET Core SDK (reflecting any global.json): Version: 3.1.301 Commit: 7feb845744

Runtime Environment: OS Name: ubuntu OS Version: 20.04 OS Platform: Linux RID: linux-x64 Base Path: /usr/share/dotnet/sdk/3.1.301/

Host (useful for support): Version: 3.1.5 Commit: 65cd789777

.NET Core SDKs installed: 3.1.301 [/usr/share/dotnet/sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

[arcaartem]

I can replicate the issue on MacOS Catalina too.

.net version installed: 3.1.403

[1ma]

Still reproducible on .NET 5.0

marcel@home:~/test$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100
 Commit:    5044b93829

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.100/

Host (useful for support):
  Version: 5.0.0
  Commit:  cf258a14b7

.NET SDKs installed:
  5.0.100 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

marcel@home:~/test$ dotnet new classlib
The template "Class library" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /home/marcel/test/test.csproj...
  Determining projects to restore...
  Restored /home/marcel/test/test.csproj (in 64 ms).
Restore succeeded.

marcel@home:~/test$ file Class1.cs 
Class1.cs: C++ source, UTF-8 Unicode (with BOM) text, with CRLF line terminators

[keegandent]

Still reproducible in .NET 6 Preview 3. Worse yet, it even uses \ path separators. Check how ExProj is included in ExSln in my below example. This really makes it seem like Unix/Linux is a second-class citizen for .NET and that changing it is low priority.

[keegan@t450s Projects]$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.100-preview.3.21202.5
 Commit:    aee38a6dd4

Runtime Environment:
 OS Name:     fedora
 OS Version:  33
 OS Platform: Linux
 RID:         fedora.33-x64
 Base Path:   /home/keegan/dotnet/sdk/6.0.100-preview.3.21202.5/

Host (useful for support):
  Version: 6.0.0-preview.3.21201.4
  Commit:  236cb21e3c

.NET SDKs installed:
  6.0.100-preview.3.21202.5 [/home/keegan/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.0-preview.3.21201.13 [/home/keegan/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.0-preview.3.21201.4 [/home/keegan/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

[keegan@t450s Projects]$ mkdir ExSln
[keegan@t450s Projects]$ cd !$
cd ExSln
[keegan@t450s ExSln]$ dotnet new sln
The template "Solution File" was created successfully.
[keegan@t450s ExSln]$ mkdir -p src/ExProj
[keegan@t450s ExSln]$ cd !$
cd src/ExProj
[keegan@t450s ExProj]$ dotnet new classlib
The template "Class Library" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /home/keegan/Projects/ExSln/src/ExProj/ExProj.csproj...
  Determining projects to restore...
  Restored /home/keegan/Projects/ExSln/src/ExProj/ExProj.csproj (in 64 ms).
Restore succeeded.

[keegan@t450s ExProj]$ cd -
/home/keegan/Projects/ExSln
[keegan@t450s ExSln]$ dotnet sln add src/ExProj/
Project `src/ExProj/ExProj.csproj` added to the solution.
[keegan@t450s ExSln]$ file ExSln.sln 
ExSln.sln: UTF-8 Unicode (with BOM) text, with CRLF line terminators
[keegan@t450s ExSln]$ cat !$
cat ExSln.sln

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.6.30114.105
MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{979BB498-A893-42EC-B4D7-CBA461FAFAE9}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ExProj", "src\ExProj\ExProj.csproj", "{D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}"
EndProject
Global
    GlobalSection(SolutionConfigurationPlatforms) = preSolution
        Debug|Any CPU = Debug|Any CPU
        Debug|x64 = Debug|x64
        Debug|x86 = Debug|x86
        Release|Any CPU = Release|Any CPU
        Release|x64 = Release|x64
        Release|x86 = Release|x86
    EndGlobalSection
    GlobalSection(SolutionProperties) = preSolution
        HideSolutionNode = FALSE
    EndGlobalSection
    GlobalSection(ProjectConfigurationPlatforms) = postSolution
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|Any CPU.Build.0 = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|x64.ActiveCfg = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|x64.Build.0 = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|x86.ActiveCfg = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Debug|x86.Build.0 = Debug|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|Any CPU.ActiveCfg = Release|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|Any CPU.Build.0 = Release|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|x64.ActiveCfg = Release|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|x64.Build.0 = Release|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|x86.ActiveCfg = Release|Any CPU
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6}.Release|x86.Build.0 = Release|Any CPU
    EndGlobalSection
    GlobalSection(NestedProjects) = preSolution
        {D6CA4D2E-E7FF-48B9-B433-921E9B4BB4E6} = {979BB498-A893-42EC-B4D7-CBA461FAFAE9}
    EndGlobalSection
EndGlobal
[keegan@t450s ExSln]$ 

[DavidKarlas]

@keegandent notice .sln file format is Visual Studio file format which was in past Windows only and as far as I know doesn't work with parsing / as path separator, but all tools in ecosystem that read .sln file format on unix systems understand that and handle that correctly.

When it comes to line endings its up to template author to decide which line ending which file will be, we should probably reconsider default for console, library and other SDK provided templates to switch to LF...

Engine itself could provide is ability for template author to specify for which files in template line ending can change and then engine changes line endings based on operating system + add flag for user to override that for example dotnet new console --line-eding=crlf...

But notice that we can't just enable this for all files, because for example some file formats/extension must stay in CRLF format I think .sln is one example of that.

[keegandent]

@DavidKarlas For the record the *.cs files are also CRLF.

Changing the *.sln and *.csproj files to Unix line endings and path separators does not seem to stop them from functioning, at least on Linux and using the CLI, VSCode, and/or MonoDevelop.

[laktak]

But notice that we can't just enable this for all files, because for example some file formats/extension must stay in CRLF format I think .sln is one example of that.

Wouldn't it make more sense to fix the tools instead of keeping legacy baggage around?

[arcaartem]

When it comes to line endings its up to template author to decide which line ending which file will be, we should probably reconsider default for console, library and other SDK provided templates to switch to LF...

Wouldn’t that create issues for Windows users in that case? (i.e. changing default to LF)

Engine itself could provide is ability for template author to specify for which files in template line ending can change and then engine changes line endings based on operating system + add flag for user to override that for example dotnet new console --line-eding=crlf...

Sounds sensible

But notice that we can't just enable this for all files, because for example some file formats/extension must stay in CRLF format I think .sln is one example of that.

Why should they stay in that format?

[DavidKarlas]

Notice that bug is open and we plan to fix this, question is just how

[keegandent]

Well the "poor man's" fix would be to generate the solution or project template as normal, but run the human-readable files through a dos2unix-esque step. This still would not correct the path separators. What I find interesting is that in the standard output for most dotnet CLI commands, the paths use *nix separators.

[laktak]

I don't think that solution would be too bad if you also run it through dotnet-format to get a consistent coding style (though dotnet-format still has some gaps).

[DavidKarlas]

Just wanted to mention we didn't forget about this issue and I found nice example why we can't just blindly convert \n to \r\n and vice-versa.

Lets assume that template has 2 files:

• build.bat(for Windows)
• build.sh(for Unix)

Lets say that user creates this template on Windows, what will end up happening is that build.sh will have \r\n. Later when someone on Linux downloads this project and tries to use build.sh it will fail with bash: ./build.sh: /bin/bash^M: bad interpreter: No such file or directory. Hence for .sh files we need to disable this functionality, hence we need to introduce new syntax for template authors to have ability to disable this automatic converting for some files in template.

[laktak]

Is it really necessary to keep \r\n at all?

I know that most (all?) tools on Windows will work with \n (this includes the shell and common utilities like Notepad).

[meyntony]

This is still a problem, may be a postAction to change line endings should be added as an action, if this cannot still be solved?

[meyntony]

I managed to fix it with the tr command, but am not able to get it to work with a postAction, can anyone spot what I'm doing wrong?

{
            "actionId": "3A7C4B45-1F5D-4A30-959A-51B88E82B5D2",
            "condition": "(OS != \"Windows_NT\")",
            "args": {
                "executable": "tr",
                "args": "-d \"\\r\" < setup.sh > set_up.sh",
                "redirectStandardOutput": "false"
            },
            "manualInstructions": [
                {
                    "text": "Run 'tr -d \"\\r\" < setup.sh > set_up.sh'"
                }
            ],
            "continueOnError": false,
            "description ": "Remove Windows line ending with from setup.sh and create a new file set_up.sh"
        }

I get the following error, but running the command directly on the CLI works fine.

[DavidKarlas]

@meyntony You don't need to do any of this, just always have \n for .sh file and no need to do any of this... Just ensure when you create NuGet package that .sh file inside has \n line endings and then users on all platforms will have \n...

[baronfel]

I did some digging in the yeoman generator ecosystem, and a common way for those generators to handle this is to also scaffold out a .gitattributes file with the following content to ensure line normalization:

* text=auto

This isn't something we should encourage for item templates, or project templates that are created in the context of a larger solution, because you'd end up with a patchwork of .gitattributes files. I think the best solution is what @DavidKarlas said before - add syntax to the sources node to allow users to opt in (or out?) of normalization.

[vlada-shubina]

We already have SpecialOperationConfig processing defined per file type. There is default set of file types with their configuration defined (operations to be taken when generating content), also it is possible to extend it via template.json configuration.

One of possible way to solve it is to add new configuration that represents line ending behavior for each file type and to configuration. This line ending configuration will be respected when generating file content (it might need another operation defined for that).

More info about it is available on wiki.

SpecialOperationConfig will be translated to operations available when content is being generated https://github.com/dotnet/templating/tree/main/src/Microsoft.TemplateEngine.Core/Operations. It might be possible to process line endings using existing operations, or we might need to create a new one.

[RehanSaeed]

I think ideally dotnet new should output files in CRLF or LF depending on OS.

In relation to git, the ideal way to handle it is to use a .gitignore file as @baronfel says. Read more about it in .gitattributes best practices.

There are a few files (.sh, .cmd, .bat) which are operating system specific and always need to be LF or CRLF (this is talked about in the link above). If this could be made extensible as @vlada-shubina suggests, I think that'd be ideal.

[baronfel]

@GrabYourPitchforks mentioned today that there's a new API in .NET 6 that might be useful here: https://learn.microsoft.com/en-us/dotnet/api/system.string.replacelineendings?view=net-6.0

[WhiteBlackGoose]

Hitting this issue. It is indeed important to have consistency. Console.WriteLine and mutliline string literals respect the platform, I'd like to expect the same from SDK. Thanks for keeping it up!

[vlada-shubina]

discussed with @baronfel, there are 2 approaches to implement this feature:

• extend SpecialOperationConfig as explained in https://github.com/dotnet/templating/issues/1248#issuecomment-1102823809 (3-4 weeks of work)
• https://github.com/dotnet/templating/issues/5505 might fix the issue too, but performance heavy.

1st approach is preferred, but need to be planned accordingly as it's a big chunk of work.

[ylluminate]

Are we any closer here? I have to say this is very frustrating on macOS and requires the creation and use of a script that detects and converts files via dos2unix. While I know this is not the case, this "feels" lazy and inconsiderate on Microsoft's part towards non-Windows systems...

[rjhind]

Yes this is something that really does need sorting. Creating a blazer or Maui app on Mac creates a lot of files that you have to cover before committing to git. Really frustrating.

[ZeroAurora]

Additionally, I'd wish to see the templating produces files without BOM.