contentFiles (cs, compile) not working in NetStandard projects.

damianh commented 7 years ago

Hi, I run LibLog - a source code logging package for library developers. I am failing to achieve NetStandard support ( https://github.com/damianh/LibLog/pull/125 )

I was unsure whether to log issue here or dotnet/sdk. I'm following https://twitter.com/davkean/status/841619178050150405 advice.

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): Package Manager Console

NuGet version (x.x.x.xxx): 4.0.0

dotnet.exe --version (if appropriate):

VS version (if appropriate): 2017

OS version (i.e. win10 v1607 (14393.321)): 10.0.14393 Build 14393

Detailed repro steps so we can see the same problem

Checkout this branch https://github.com/damianh/LibLog/tree/loqu8-netstandard
To build a nuget package, run .\build.ps1. This will put a package in build directory
Open the solution src\ContentFilesTest\ContentFilesTest.sln in VS2017. You will see two projects that don't compile where we will install LibLog.
In Package Manager Console, select the project DotNetFramework46 and install LibLog: install-package LibLog -prelease -source ../../build/ . LibLog should be installed into the project DotNetFramework46, visible in folder App_Packages\... and that project should now compile.
In Package Manager Console, select the project DotNetStandard and install LibLog: install-package LibLog -prelease -source ../../build/. The project DotNetStandard continues to not compile and the type LogProvider cannot be resolved.

Other suggested things

There is more information in the comments in the related issue https://github.com/damianh/LibLog/pull/125 , including the fact that no transformed file is written to obj\Debug\...

Verbose Logs

https://github.com/damianh/LibLog/pull/125#issuecomment-286391120

Sample Project

Let me know if you want a zip of the project instead of checking out the branch.

emgarten commented 7 years ago

Packages.config projects use the content folder Project.json/PackageReference/NETCore SDK projects use the contentFiles folder

In your steps the DotNetFramework46 project ends up being a packages.config project which does not use the contentFiles folder where your pp file is.

To allow your package to work in both types of projects you should put the pp file in both places.

The reason for this is that older packages.config projects require Visual Studio to install packages, and items from the content folder are actually copied into the project itself at install time. With project.json/PackageReference packages can change at restore time so the contentFiles folder is used which only contains immutable files.

damianh commented 7 years ago

Thanks or the response @emgarten , however I'm not grokking what I should actually do.

To allow your package to work in both types of projects you should put the pp file in both places.

It is as far as I can see... if not what changes are need to https://github.com/damianh/LibLog/pull/125/files#diff-6ec70a9e9835da1a3387257b2d950192R36 to make this work?

Cheers.

tofutim commented 7 years ago

Loqu8.LibLog.4.2.9.nupkg.zip

Try this

damianh commented 7 years ago

@tofutim I'll respond on https://github.com/damianh/LibLog/pull/125

emgarten commented 7 years ago

@damianh you need to have the files under both content and contentFiles.

damianh commented 7 years ago

@emgarten Like this?

<package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
    <metadata minClientVersion="3.3.0">
        <...>
        <dependencies>
            <group targetFramework=".NetStandard1.1">
                <dependency id="Microsoft.CSharp" version="[4.0.1, )" />
                <dependency id="System.Dynamic.Runtime" version="[4.0.11, )" />
            </group>
        </dependencies>
        <contentFiles>
            <files include="contentFiles\cs\**\*.*" buildAction="Compile" />
        </contentFiles>
    </metadata>
    <files>
        <file src="LibLog.cs.pp" target="content\App_Packages\LibLog.4.2\LibLog.cs.pp" />
        <file src="LibLog.cs.pp" target="contentFiles\App_Packages\LibLog.4.2\LibLog.cs.pp" />
    </files>
</package>

damianh commented 7 years ago

you need to have the files under both content and contentFiles.

Unfortunately this statement is confusing me:

There is a <contentFiles> tag, do you mean that?
There is a contentFiles\ path in the <files> element under <contentFiles>, do yo meant that?

Honestly a working nuspec would be much easier to move forward on.

damianh commented 7 years ago

    <files>
      <file src="LibLog.cs.pp" target="content\App_Packages\LibLog.4.2\LibLog.cs.pp" />
      <file src="LibLog.cs.pp" target="contentFiles\App_Packages\LibLog.4.2\LibLog.cs.pp" />
    </files>

... did not work.

emgarten commented 7 years ago

Files at the root of contentFiles are not picked up. You need to have:

<files>
      <file src="LibLog.cs.pp" target="content\App_Packages\LibLog.4.2\LibLog.cs.pp" />
      <file src="LibLog.cs.pp" target="contentFiles\cs\any\App_Packages\LibLog.4.2\LibLog.cs.pp" />
    </files>

Your nuspec contentFiles section has contentFiles\cs\** which doesn't match contentFiles\App_Packages where you are putting the file.

damianh commented 7 years ago

Applied your suggest (please correct if wrong). Doesn't appear to work.

    ....
        <contentFiles>
            <files include="contentFiles\App_Packages\**\*.*" buildAction="Compile" />
        </contentFiles>
    </metadata>
    <files>
      <file src="LibLog.cs.pp" target="content\App_Packages\LibLog.4.2\LibLog.cs.pp" />
      <file src="LibLog.cs.pp" target="contentFiles\cs\any\App_Packages\LibLog.4.2\LibLog.cs.pp" />
    </files>

emgarten commented 7 years ago

@damianh take a look at https://www.nuget.org/packages/ContentFilesExample/

damianh commented 7 years ago

@emgarten Thank you very much for that package! However, I've been testing it and there seems to be a number of issues. I hope the problem is me :(

1. Installing the package doesn't show the content files as part of the project:

Closing and re-opening the solution subsequently shows the content files:

2. The files `ExampleInternals.cs` and `ExampleReader.cs.pp` are nowhere to be seen and their types are not resolvable:

3. `uninstall-package` doesn't remove the added files:

Closing and re-opening the solution subsequently shows the files removed.

emgarten commented 7 years ago

Thanks for the details @damianh

@natidea has anything changed with .pp files? I tried also and didn't see them getting generated in VS 2017 RTM but I do see them in project.assets.json.

The delay in adding/removing the files from Solution Explorer looks like a VS issue, I see them written out correctly to the obj props file.

bmeredith commented 7 years ago

I was also able to reproduce the same issues @damianh mentioned above when trying to use the ContentFilesExample package.

damianh commented 7 years ago

I am relieved that it has been repro'd. Thanks guys.

damianh commented 7 years ago

@emgarten @natidea Gents, any update or thoughts on this? It appears that source transforms are completely broken. LiteGuard is another project that is affected. cc @adamralph

adamralph commented 7 years ago

LiteGuard, due to it's simplicity, is also a great test-bed for new/changed tooling. I've used it personally as my "hello world" for learning about PCL's, multi-targetting, project.json, etc. etc., with the benefit that it's also a real package, with real users. I'd be more than happy to use LiteGuard to help smoke test any work that comes out of this issue, from a package producer POV, .

blackcity commented 7 years ago

Come on NuGet guy's. This problem has been reported at least one year ago. ContentFiles did not work in project.json and also do not work in the new csproj based projects. We need at least a way to copy additional assets to the project. This can't be too hard.

emgarten commented 7 years ago

@blackcity this issue is currently narrowed down to .pp files not being transformed for NETCore projects. If you are unable to get contentFiles to work at all then you may have a package that is authored incorrectly. Would you own a new issue explaining the issues you are currently seeing along with repro steps and a copy of the package that doesn't work?

blackcity commented 7 years ago

It is exactly like described in this post: https://github.com/NuGet/Home/issues/4803#issuecomment-287523738

1) Cannot see the files/folders in Solution Explorer after the package has been installed. 2) Closing and reopening the solution shows files/folders (folders with a red cross) but 3) Files/folders are not physically copied to the solution (only referenced?) 4) Uninstalling leaves orphaned entries wich results in errors.

For our packages it is important that the content files (assets) are copied to the solution, not just referenced. With the package we need to copy config files (XML files) the developer wants to change. This was standard in the packages.config era.

We cannot publish a package with this strange behaviour without our customers killing us.

Btw: Just tested again with ASP.NET Core Project (.NetCoreApp1.1) and the package mentioned above: https://www.nuget.org/packages/ContentFilesExample/

emgarten commented 7 years ago

For our packages it is important that the content files (assets) are copied to the solution, not just referenced. With the package we need to copy config files (XML files) the developer wants to change. This was standard in the packages.config era.

This is not supported with project.json/PackageReference. With packages.config users had to run install and uninstall commands explicitly and this was done through Visual Studio.

With project.json/PackageReference packages can float which allows them to change from restore to restore without an explicit install/uninstall action being performed. For this reason the contentFiles folder is immutable and cannot carry a state like the packages.config content folder.

The plan going forward for this is to allow users to move files from under the contentFiles folder into their project through a gesture in Visual Studio. Currently the include of these files is written out to the auto generated nuget props file in the obj folder which allows them to be removed or overridden.

My suggestion for copying config XML files to the user project is to handle this in an init.ps1 script, or possibly in a .targets file.

blackcity commented 7 years ago

Thanks @emgarten for this detailed answer. So init.ps1 and and targets work in ASP.NET Core projects (csproj)? Currently it is a bit confusing because of the lack of docs. But I'll try it out.

emgarten commented 7 years ago

init.ps1 and and targets work in ASP.NET Core projects (csproj)?

Yes, if installed using Visual Studio init.ps1 will be executed. This works on all types of projects. The script will also run when opening the solution if the nuget powershell console is open.

If someone manually edits the csproj outside of Visual Studio and restores then it will not run, but you could catch this scenario in a targets file where the config is missing and fail the build with a helpful message.

natidea commented 7 years ago

Sorry just getting to this after my vacation. I took a look at the pp file transformation issue and found a couple of things:

For the ContentFilesExample package, I don't see pp files listed as dependencies in the assets file. They occur in the "libraries" but we are only processing content from the active TFM here: Microsoft.PackageDependencyResolution.targets#L213
For the LiteGuard package, I do see the pp files in the dependencies, but they were not being processed because CopyLocalLockFileAssemblies=false here: Microsoft.PackageDependencyResolution.targets#L414 I was able to fix this by setting the property in my project file:
```
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
```
Now I can use the Guard.

So somewhere along the way, some concepts may have been crossed. We default CopyLocalLockFileAssemblies to true in that target file, but it gets set to false in Microsoft.NET.Sdk.BeforeCommon.targets#L57. Perhaps the real issue is that the produce content assets task is tied to ResolveLockFileCopyLocalProjectDeps and should instead be triggered in some other way.

Adding @livarcocc and @dsplaisted from SDK who would need to resolve this.

emgarten commented 7 years ago

I should have mentioned, I had to modify ContentFilesExample locally to add the netstandard TFM, this package was originally created for a UWP tutorial. I'll take a look at updating it on nuget.org soon.

blackcity commented 7 years ago

@emgarten This seems to work now. We are using init.ps1, build props and targets. Last question: We have a NuGet package with 3 TFM (net45, net461, netstandard1.6):

lib
   net45
       library.dll    // for traditional Framework
   net461
       library.dll    // for ASP.NET Core (full Framework)
   netstandard1.6
       library.dll    // for ASP.NET Core

The net461 library can only run in ASP.NET Core projects (the full Framework). Is there any way to prevent traditional projects from referencing the net461 TFM and instead reference the net45 assembly?

emgarten commented 7 years ago

Is there any way to prevent traditional projects from referencing the net461 TFM and instead reference the net45 assembly?

NuGet will take the highest compatible asset. If the project is net462 for example it will take net461. If the project is net46 it will use net45 since net461 is not compatible. If you use a targets file in the package instead of the lib folder you can select the dll based on other project properties instead.

blackcity commented 7 years ago

Great answer thanks. Really last question: What if the project is net462 and the NuGet package provides TFM for netstandard1.5 and net461. Does the project choose the netstandard1.5 library because it is the highest compatible TFM or does it choose net461 because it is more specific. In your tests we added a netstandard1.4 TFM as replacement for net461, but the project choosed the net45 assembly which isn't compatible with ASP.NET Core.

emgarten commented 7 years ago

If the project is net* then a compatible net* from the package will be used. If there are no compatible matches then it will fallback to netstandard. It is debatable which should be used, but NuGet currently favors keeping the same framework as the project.

blackcity commented 7 years ago

@emgarten Thanks for your fantastic support.

Two suggestions. 1) Please provide docs! NuGet is essential for .NET/.NET Core development. Imagine how many developers struggle when it comes to more advanced things like content files, install scripts, build props and targets. There is almost no documentation out there. We have to read through blogs, outdated docs or ask you guys stupid questions. What a waste of time! Our time and your time. We need maybe 20 pages of in-depth coverage. That's all. Buy me a beer and I'll do it for you. 2) Add a TFM for libraries that can only be executed in .NET Core over full Framework environments. In the past you had these TFMs like dnx or netcore. If we have a library that can only run on .NET Core over full Framework we cannot just put it in a lib/net folder because it could be imported into traditional projects. Why not just reactivate netcore (preferred) or maybe add an attribute like this?

     <dependencies>
       <group targetFramework=".NETFramework4.6" netcore="true"> 
         <dependency id="SomeLibrary" version="1.0.0" /> 
       </group> 
     </dependencies>

emgarten commented 7 years ago

Moving this to https://github.com/dotnet/sdk/issues/1100

The issue here appears to be in the SDK. I see the correct outputs from NuGet.

NuGet / Home