dotnet / sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
https://dot.net/core
MIT License
2.69k stars 1.06k forks source link

dotnet pack always failed on Ubuntu-18.04 with dotnet-sdk-3.0-preview3 #3024

Open itn3000 opened 5 years ago

itn3000 commented 5 years ago

Overview

dotnet pack command always failed in Ubuntu-18.04 in dotnet-sdk-3.0-preview3. if dotnet-sdk-2.2 was used, it successed.

Environment

platform is ubuntu-18.04-x64(created from azure virtual machine)

here is my dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview3-010431
 Commit:    d72abce213

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /home/itn/dotnet/sdk/3.0.100-preview3-010431/

Host (useful for support):
  Version: 3.0.0-preview3-27503-5
  Commit:  3844df9537

.NET Core SDKs installed:
  3.0.100-preview3-010431 [/home/itn/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview3-19153-02 [/home/itn/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview3-27503-5 [/home/itn/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Steps to reproduce

  1. download dotnet-sdk-3.0-preview3 for linux-x64 from SDK's download page
  2. extract archive to directory($HOME/dotnet)
  3. add $HOME/dotnet to $PATH env
  4. run dotnet new classlib --name test1
  5. change directory to test1
  6. change TargetFramework to netstandard2.1
  7. run dotnet pack

Expected Behavior

test1.nupkg was created in bin directory.

Actual Behavior

failed to create nupkg with following error message.

/home/itn/dotnet/sdk/3.0.100-preview3-010431/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(199,5): error NU5019: File not found: '/home/itn/test1/bin/Debug/netstandard2.1/test1.dll'. [/home/itn/test1/test1.csproj]

/home/itn/test1/bin/Debug/netstandard2.1/test1.dll was created successfuly. I also tried to change TargetFramework to netstandard2.0, but did not success.

itn3000 commented 5 years ago

and here is my msbuild's binary log dotnet-pack-failed.zip

livarcocc commented 5 years ago

@peterhuene can you try that on your ubuntu box?

peterhuene commented 5 years ago

I have an Ubuntu box? 😄 I'm actually an Arch user. I can spin up an Ubuntu VM to investigate, though.

livarcocc commented 5 years ago

@peterhuene Sorry, I actually thought you used Ubuntu.

peterhuene commented 5 years ago

I can't reproduce it with Arch. Going to spin up a VM.

itn3000 commented 5 years ago

oh, I'm sorry. I setup another VM(conditions are same), dotnet pack successed. It may be environment-specific problem. Should the issue be closed?

peterhuene commented 5 years ago

I've also been unable to reproduce. Does this consistently happen with the original VM if you try with another project?

I'd like to figure out just a little more before closing.

itn3000 commented 5 years ago

It can be reproduced in another project(like console) on same machine.

peterhuene commented 5 years ago

If you dotnet build prior to dotnet pack, does it succeed?

Shot in the dark: do you happen to have an environment variable named NoBuild set?

itn3000 commented 5 years ago

I tried dotnetb build then dotnet pack, build was successed but pack was failed. NoBuild seemed to be set.here is my export command output.

declare -x DOTNET_ROOT="/home/itn/dotnet"
declare -x HOME="/home/itn"
declare -x LANG="C.UTF-8"
declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
declare -x LESSOPEN="| /usr/bin/lesspipe %s"
declare -x LOGNAME="itn"
declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:"
declare -x MAIL="/var/mail/itn"
declare -x OLDPWD
declare -x PATH="/home/itn/dotnet:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/itn/.dotnet/tools"
declare -x PWD="/home/itn"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_CLIENT="27.82.186.113 52981 22"
declare -x SSH_CONNECTION="27.82.186.113 52981 10.1.1.4 22"
declare -x SSH_TTY="/dev/pts/0"
declare -x TERM="xterm"
declare -x USER="itn"
declare -x XDG_DATA_DIRS="/usr/local/share:/usr/share:/var/lib/snapd/desktop"
declare -x XDG_RUNTIME_DIR="/run/user/1000"
declare -x XDG_SESSION_ID="3"
per-samuelsson commented 5 years ago

I experience this too.

Under WSL, with this env:

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview6-012264
 Commit:    be3f0c1a03

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /home/per/dotnet/sdk/3.0.100-preview6-012264/

Host (useful for support):
  Version: 3.0.0-preview6-27804-01
  Commit:  fdf81c6faf

.NET Core SDKs installed:
  3.0.100-preview6-012264 [/home/per/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview6.19307.2 [/home/per/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview6-27804-01 [/home/per/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

I've tried on different projects. One is a library targeting netstandard2.0. The other a tool/exe targeting netcoreapp3.0. Both show similar behavior, reporting error such as:

/home/per/dotnet/sdk/3.0.100-preview6-012264/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error NU5019: File not found: '/home/per/Git/Starcounter/Starcounter.Nova/src/Starcounter.Nova/bin/Debug/netstandard2.0/Starcounter.Nova.dll'. [/home/per/Git/Starcounter/Starcounter.Nova/src/Starcounter.Nova/Starcounter.Nova.csproj]

In both cases, dotnet build works, and the files being complained about are there (at least when the command finishes). Doing dotnet build first and then doing --no-build changes nothing, and neither does using --force.

Edit: And oh, the same tooling + code works well to pack on Windows.

per-samuelsson commented 5 years ago

Since this is blocking a beta-release of ours, I decided to strip as much as possible to see if I could find a way around this. I've found it's reproducible every time even on a new, empty project.

mkdir app1
cd app1
dotnet new console
dotnet pack

Microsoft (R) Build Engine version 16.2.0-preview-19278-01+d635043bd for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 17.19 ms for /home/per/proj/app1/app1.csproj.
/home/per/dotnet/sdk/3.0.100-preview6-012264/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(158,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [/home/per/proj/app1/app1.csproj]
  app1 -> /home/per/proj/app1/bin/Debug/netcoreapp3.0/app1.dll
/home/per/dotnet/sdk/3.0.100-preview6-012264/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error NU5019: File not found: '/home/per/proj/app1/bin/Debug/netcoreapp3.0/app1.runtimeconfig.json'. [/home/per/proj/app1/app1.csproj]

The env is still the same as post above.

A bit surprised such a basic thing passed acceptance tests, even though I know this is a preview? Is there any immediate plans to fix this?

We release our product as a set of packages and need to build native bits on Linux, so we are pretty blocked on this, or need to move built bits from Linux to Windows agents to do the packaging, which is of course something we would rather not want to spend time on.

miyconst commented 5 years ago

Another reproduction:

dotnet new ConsoleDemo
dotnet build
dotnet run
dotnet pack
Microsoft (R) Build Engine version 16.2.0-preview-19278-01+d635043bd for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 12.47 ms for /home/star/Documents/ConsoleDemo/ConsoleDemo.csproj.
/home/star/dotnet/sdk/3.0.100-preview6-012264/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(158,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [/home/star/Documents/ConsoleDemo/ConsoleDemo.csproj]
  ConsoleDemo -> /home/star/Documents/ConsoleDemo/bin/Debug/netcoreapp3.0/ConsoleDemo.dll
/home/star/dotnet/sdk/3.0.100-preview6-012264/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error NU5019: File not found: '/home/star/Documents/ConsoleDemo/bin/Debug/netcoreapp3.0/ConsoleDemo.runtimeconfig.json'. [/home/star/Documents/ConsoleDemo/ConsoleDemo.csproj]

The /home/star/Documents/ConsoleDemo/bin/Debug/netcoreapp3.0/ConsoleDemo.runtimeconfig.json file exists.

Content of `ConsoleDemo.runtimeconfig.json` ```json { "runtimeOptions": { "tfm": "netcoreapp3.0", "framework": { "name": "Microsoft.NETCore.App", "version": "3.0.0-preview6-27804-01" } } } ```
`dotnet --info` ``` .NET Core SDK (reflecting any global.json): Version: 3.0.100-preview6-012264 Commit: be3f0c1a03 Runtime Environment: OS Name: ubuntu OS Version: 18.04 OS Platform: Linux RID: ubuntu.18.04-x64 Base Path: /home/star/dotnet/sdk/3.0.100-preview6-012264/ Host (useful for support): Version: 3.0.0-preview6-27804-01 Commit: fdf81c6faf .NET Core SDKs installed: 3.0.100-preview6-012264 [/home/star/dotnet/sdk] .NET Core runtimes installed: Microsoft.AspNetCore.App 3.0.0-preview6.19307.2 [/home/star/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.0.0-preview6-27804-01 [/home/star/dotnet/shared/Microsoft.NETCore.App] To install additional .NET Core runtimes or SDKs: https://aka.ms/dotnet-download ```

dotnet-pack-diagnostic.zip

miyconst commented 5 years ago

Okay, this is funny, but I have nailed down the issue. It's a bug in System.Text.RegularExpressions on WSL.

The applicaiton

using System;
using System.Text.RegularExpressions;

namespace ConsoleDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            string test = @"Parent/Child.Grand.Child";
            Regex reg = new Regex('^' + Regex.Escape(test) + '$', RegexOptions.IgnoreCase);

            Console.WriteLine("Test string: {0}", test);
            Console.WriteLine("Regex: {0}", reg);
            Console.WriteLine("Regex.IsMatch: {0}", reg.IsMatch(test));
        }
    }
}

Output on vanilla Ubuntu 18.04

Test string: Parent/Child.Grand.Child
Regex: ^Parent/Child\.Grand\.Child$
Regex.IsMatch: True

Output on WSL Ubuntu 18.04

Test string: Parent/Child.Grand.Child
Regex: ^Parent/Child\.Grand\.Child$
Regex.IsMatch: False

Removing RegexOptions.IgnoreCase option fixes the output on WSL as well.

Why this affects dotnet pack

NuGet is filtering files by the exclude parameter using Regex here:

Which results into file not found exceptions, since no files pass the regex filter.

Apart of fixing the regex issue, NuGet shall not throw FileNotFound exception if the file exists, but filtered out by the exclusions.

alexrp commented 5 years ago

Is there a known workaround for this? I'm getting the same issue under WSL 2.

miyconst commented 5 years ago

Is there a known workaround for this? I'm getting the same issue under WSL 2.

As pointed out by @stephentoub, the issue can be solved by manually setting the LANG environment variable.

export LANG=en-US.UTF-8
danmoseley commented 5 years ago

See https://github.com/NuGet/Home/issues/8246 also