Support for piping the output of `dotnet sln list`

natemcmaster commented 5 years ago

I would like to use dotnet sln list in a script to automate some actions on projects. Unfortunately, the output of dotnet sln list is not suitable for piping into a script variable because it contains extraneous information.

Steps to reproduce

# powershell
$projectFiles = dotnet sln list

# bash
project_files=$(dotnet sln list)

Expected behavior

dotnet sln list should only produce output that lists the files, or support a flag like dotnet sln list --porcelain which suppresses the other info that breaks scripting.

Actual behavior

The first two lines of output are always irrelevant, and one first use, the output contains a welcome message that cannot be suppressed.

$ dotnet sln list

Welcome to .NET Core!
---------------------
Learn more about .NET Core: https://aka.ms/dotnet-docs
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
Project(s)
----------
Abstractions/src/Microsoft.AspNetCore.DataProtection.Abstractions.csproj
Abstractions/test/Microsoft.AspNetCore.DataProtection.Abstractions.Tests.csproj
AzureKeyVault/src/Microsoft.AspNetCore.DataProtection.AzureKeyVault.csproj

Environment data

dotnet --info output:

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview-010184
 Commit:    c57bde4593

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.14
 OS Platform: Darwin
 RID:         osx.10.14-x64
 Base Path:   /Users/namc/src/aspnet/AspNetCore/.dotnet/sdk/3.0.100-preview-010184/

Host (useful for support):
  Version: 3.0.0-preview3-27414-8
  Commit:  2c740c0ea8

.NET Core SDKs installed:
  3.0.100-preview-010184 [/Users/namc/src/aspnet/AspNetCore/.dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview-19075-0444 [/Users/namc/src/aspnet/AspNetCore/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.1 [/Users/namc/src/aspnet/AspNetCore/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview-27324-5 [/Users/namc/src/aspnet/AspNetCore/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview3-27414-8 [/Users/namc/src/aspnet/AspNetCore/.dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

livarcocc commented 5 years ago

This portion is there only for the first time and for legal reasons:

Welcome to .NET Core!
---------------------
Learn more about .NET Core: https://aka.ms/dotnet-docs
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

I don't believe we can or willl get rid of it for the first run.

That would leave you with

Project(s)
----------
Abstractions/src/Microsoft.AspNetCore.DataProtection.Abstractions.csproj
Abstractions/test/Microsoft.AspNetCore.DataProtection.Abstractions.Tests.csproj
AzureKeyVault/src/Microsoft.AspNetCore.DataProtection.AzureKeyVault.csproj

Isn't this something that you could handle in a script?

natemcmaster commented 5 years ago

Shouldn't DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 suppress the first part? We have that set on all CI builds to avoid these kinds of issues.

Isn't this something that you could handle in a script?

Sure, and we do, but it's a best guess and is apparently fragile (just failed our CI a few times). I'm still interested in a flag or env var I can set so that I don't have to filter the output based on a guess about whether the output is supposed to be a file path or not.

walkermundo commented 5 years ago

I too would like an option to the dotnet sln list command to suppress the first two lines. It would be much cleaner to know that all output is usable and to not special case ignoring certain lines.

wli3 commented 5 years ago

However we need to display the message. Does run the same command twice and let the first one finish displaying first run message an option? Or add a command that does no op but only show the first run message?

@KathleenDollard

KathleenDollard commented 5 years ago

There are three things here:

DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 isn't the documented or right way to suppress the welcome message and telemetry. Will can you add that hear for anyone with @natemcmaster problem
Sometime past 3.0 I would like to add options for output on all commands. That would be much easier if we have System.CommandLine (and it's rendering) and we do not have a schedule for that.
If dropping the first two lines of the output is problematic, is there a verbosity level we could use to indicate no headers in all of our lists?

wli3 commented 5 years ago

Sometime past 3.0 I would like to add options for output on all commands. That would be much easier if we have System.CommandLine (and it's rendering) and we do not have a schedule for that.

Please add me to the loop. I know some command line app do that, but it is still strange. Especially on Unix. Plain text is the format for data exchange. And Powershell try to change that by using objects.

If dropping the first two lines of the output is problematic, is there a verbosity level we could use to indicate no headers in all of our lists?

I like that better.

am11 commented 4 years ago

DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 isn't the documented

CI automation (even outside of dotnet org) used it to mute "Welcome to .NET" message. DOTNET_SKIP_FIRST_TIME_EXPERIENCE knob has been removed in v3.0 RC2 (7efbf4770630fd370779d222ce9aba35ffe56100).

The new. very implementation detail dependent, way for 3.0 is to create .dotnetFirstUseSentinel marker file:

#!/usr/bin/env sh

# to suppress "Welcome to .NET.." FTUE message
touch "$HOME"/.dotnet/"$(dotnet --version)".dotnetFirstUseSentinel

dotnet publish --nologo ...

aolszowka commented 4 years ago

This makes it difficult to compose new SLN files from existing SLN files.

For example I have ProjectA.sln and ProjectB.sln, I want to create ProjectC.sln which is a combination of the two. In theory I should be able to write a one-liner that pipes these right back into dotnet sln to create the composed Solution files (filtering for distinct of course).

Today because this prints out the following lines:

Project(s)
----------

Additional pre-processing must be done on the files within script (as suggested by @livarcocc ). This is noted by @walkermundo above.

Its a real bummer to have to apply this hack which is pretty much guaranteed to break when this gets fixed. Those magic numbers are a real downer.

Furthermore all of these files are relative paths, which requires additional processing. Here's a powershell snippet that we're using based on the behavior today:

function Main()
{
    Compose-Solution -SolutionName "ProjectC" -TargetSolutions @("S:\ProjectA.sln","S:\ProjectB.sln")
}

function Compose-Solution([string]$SolutionName, [string[]]$TargetSolutions)
{
    &dotnet.exe new sln -n $SolutionName

    $AllProjects = Get-AllProjects -TargetSolutions $TargetSolutions

    foreach($project in $AllProjects)
    {
        &dotnet.exe sln $SolutionName.sln add $project
    }
}

function Get-AllProjects([string[]]$TargetSolutions)
{
    foreach($solution in $TargetSolutions)
    {
        $AllProjects += Get-ProjectsInSolution($solution)
    }

    $AllProjects | Select-Object -Unique
}

function Get-ProjectsInSolution([string]$TargetSolution)
{
    $solutionDirectory = [System.IO.Path]::GetDirectoryName($TargetSolution)
    $solutionListing = &dotnet.exe sln $TargetSolution list | Select-Object -Skip 2

    foreach($project in $solutionListing )
    {
        $relativePath = [System.IO.Path]::Combine($solutionDirectory, $project)
        [System.IO.Path]::GetFullPath($relativePath)
    }
}

Main

woutervanvliet commented 4 years ago

To remove the first two lines of output, you can use this solution: https://stackoverflow.com/a/24542679/803283

dotnet sln list | sed 1,2d

Or, if you're unsure about how the output might change in the future:

dotnet sln ./build.sln list | grep -E \.csproj$

dotnet / sdk