project.assets.json does not work with shared intermediates

[davkean]

Files that generated into the intermediates, typically always include the project name in the name of the file:

Debug\ConsoleApp146.AssemblyInfo.cs
Debug\ConsoleApp146.csproj.FileListAbsolute.txt
Debug\ConsoleApp146.csprojResolveAssemblyReference.cache
Debug\ConsoleApp146.dll
Debug\ConsoleApp146.pdb
ConsoleApp146.1.0.0.nuspec
ConsoleApp146.csproj.nuget.g.props
ConsoleApp146.csproj.nuget.g.targets

The assets file does not:

project.assets.json

This is going to lead to clashes when projects share intermediate directories, which is pretty common in large projects such as Roslyn.

[rrelyea]

good issue. we think we can change without too much work...but likely too late for rtm. /cc @jainaashish @emgarten

[emgarten]

Agreed, this would be a good change. We've been working towards making this file name/path extensible, but there a few places still expecting project.assets.json.

[davkean]

Yeah, let's fix it later - just make sure you coordinate with us.

[emgarten]

$(ProjectAssetsFile) will contain the full path that NuGet writes to. This of course won't exist if the project hasn't been restored yet, so it might be hard to rely on this.

[davkean]

Good to know, we should be respecting that: https://github.com/dotnet/roslyn-project-system/issues/1437

[davkean]

Make note - this also affects two projects in the same directory: https://github.com/dotnet/roslyn-project-system/issues/1528

[caleblloyd]

When I define a different IntermediateOutputPath, nuget files still always end up in $(ProjectDir)\obj\. I am following this MSDN blog post

Example:

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.1</TargetFramework>
    <IntermediateOutputPath>$(SolutionDir)\obj\$(Configuration)\$(MSBuildProjectName)\</IntermediateOutputPath>
    <OutputPath>$(SolutionDir)\bin\$(Configuration)\$(AssemblyName)\</OutputPath>
  </PropertyGroup>

All other files get moved correctly, except for these 3 which still appear in $(ProjectDir)\obj\:

App.csproj.nuget.g.props
App.csproj.nuget.g.targets
project.assets.json

[emgarten]

~@caleblloyd NuGet reads BaseIntermediateOutputPath~

update by @nkolev92

NuGet now respects MSBuildProjectExtensionsPath.

Please refer to microsoft/msbuild#1603 and NuGet/NuGet.Client#2131 and NuGet/NuGet.Client#2121.

Original comment: https://github.com/NuGet/Home/issues/4463#issuecomment-404863327

[davkean]

Given where the base intermediate path is set, you can't set it without expanding SDK into explicit imports of a props and targets.

[caleblloyd]

I am trying to run a Docker container that has code mounted into it while also developing on the host inside of an IDE. The IDE keeps running Dotnet Restore and conflicting with the Docker container's Dotnet Restore.

I have to do multiple mounts to exclude the bin and obj folders if they are in the default path, so I thought maybe I'd move them using BaseIntermediateOutputPath. But this causes Entity Framework Tools and dotnet test to not work properly.

Is there another solution to this besides for waiting for the ability to name project.assets.json different names on the Host and in Docker? I can run different build configurations on the Host and Docker (e.g. run dotnet run -c Debug on the Host and run dotnet run -c Docker on Docker) and right now the only conflict I'm getting is that Nuget keeps clobbering the other side

[davkean]

Can you expand on the issues you have with EF and dotnet test when you change BaseIntermediateOutputPath with a bug over on http://github.com/roslyn-project-system? I'll move them to the appropriate repros if it's a bug in those tools.

[caleblloyd]

Yes, I will do that in a few days. Thanks!

[jdasilva]

@davkean Does this mean that having a shared intermediate folder is supported once these issues are fixed?

[davkean]

@jdasilva These are two issues that I found scouting it - but I've not confirmed that it's fully supported yet. Sounds like @caleblloyd was running into issues that I'd like to capture

[jdasilva]

Thanks, I was thinking about a comment like this https://github.com/dotnet/sdk/issues/760#issuecomment-276259786. I've been using a common intermediate folder for a couple of projects since at least VS2013 and this still works under VS2017 with the old project system. I'd love to move to the new one, so I really meant "supported" in the sense of if something blocks this will it be considered a bug? You seem to be doing that already which is great and I just want to make sure I'm on the same page.

[davkean]

@jdasilva Sorry missed this comment (for some reason NuGet issues go to my personal email).

Yes we'd consider anything blocking multiple projects from building to a common intermediate as a bug - this is the reason that we prefix most things in obj with the project name. This issue is the last one that I know that is blocking building to a common intermediate folder (MSBuild fixed their last one)

[davkean]

Here's another issue that results as a problem of this - https://github.com/dotnet/project-system/issues/1935. Basically, building a new csproj in the same directory as the old csproj, breaks the build because the old csproj picks up the same assets file.

[emgarten]

@davkean when will https://github.com/dotnet/project-system/issues/1437 be fixed?

[nguerrera]

Possible issues in SDK too: dotnet/sdk#1438

[nkolev92]

Some related work: https://github.com/NuGet/NuGet.Client/pull/2056

[nickrandolph]

Just got this issue where we have two csproj in same folder. One is multi-targetted (netstandard2.0, net461 etc), the other just net461. Can't resolve a reference to a net45 nuget package.

[davkean]

@emgarten If you folks are blocked on dotnet/project-system#1437 to fix this - I'll fix it immediately in 15.8.

[nkolev92]

@davkean I'm not sure if that's the only "blocker", but fixing that would certainly go a long way. //cc @rrelyea

[piksel]

For anyone else finding this and needs a workaround, save the following as Directory.Build.props in your project directory:

<Project>
  <PropertyGroup>
    <MSBuildProjectExtensionsPath>obj\$(MSBuildProjectName)\</MSBuildProjectExtensionsPath>
  </PropertyGroup>
</Project>

This will cause project.assets.json to be written to $(ProjectDir)\obj\ProjectFileWithNoExtension\project.assets.json. And works fine with multiple .csproj in the same directory (what I needed).

Comment on this gist if it's not related to the issue: https://gist.github.com/piksel/950646e301e0fd994465b26773170494

[lucasgl]

Any reason BaseIntermediateOutputPath cannot also override MSBuildProjectExtensionsPath? (I would guess ordering...)

[nkolev92]

@lucasgl

NuGet now respects MSBuildProjectExtensionsPath.

Please refer to https://github.com/Microsoft/msbuild/issues/1603 and https://github.com/NuGet/NuGet.Client/pull/2131 and https://github.com/NuGet/NuGet.Client/pull/2121.

MSBuildProjectExtensionsPath defaults to BaseIntermediateOutputPath, the trick you need to set it early enough in the evaluation to avoid these problems. Doing in the Directory.Build.Props does just that.

[BrightLight]

Glad I found this. I ran into the same issue last week. Is there any ETA on when this will be fixed? Our build server calls msbuild with /p:BaseIntermediateOutputPath=<somefolder> (a "Temp" folder inside the working directory of the build job). All was fine as long as the solution only contained .Net framework targets. Now colleagues have added two(!) .Net Standard projects which breaks the build because, of course, there can only be one project.assets.json in this "Temp" folder. Is there a temporary workaround available?

[piksel]

@BrightLight I posted a workaround

[jeffkl]

I'd like to revive this bug and work on a fix.

There is no way to guarantee that restore will result in a successful build unless the MSBuildProjectExtensionsPath is unique for each project. If you try to use a rooted path like obj\$(MSBuildProjectName) you could still have two projects in the tree with the same name. So renaming project.assets.json wouldn't help us there.

The only way to guarantee that the assets from a restore are unique per project is to have a unique path per project. So I suggest that we fail restore if multiple projects share the same directory. Otherwise you get non-deterministic restores since there's no guarantee what order the project.assets.json is written. Not to mention that you get into a state where restore succeeds but your build fails with compiler errors about missing types because package assets are wrong.

Does everyone agree that we should just block the ability for projects in restore to share MSBuildProjectExtensionsPath?

[SwooshyCueb]

With this solution, could two project files exist in the same directory if they set MSBuildProjectExtensionsPath to different locations?

[jeffkl]

@SwooshyCueb yes that would work. As long as the MSBuildProjectExtensionsPath is unique, NuGet would be restore just fine. However, MSBuild will give you an error if you build in that directory and don't specify a project to build.

[nguerrera]

I think a best effort should be made to make the path unique in common cases. Consider the trivial case of two projects in the same folder with default output paths. For ages, this simple case worked because everything put the project name in obj files, until assets file came along. So I think it is good to have the error as you suggest, but I still see value in being unique by default up to project name.

[nkolev92]

There are no technical blockers anymore afaik.

The concern would be the migration plan. For better or worse newer NuGet versions are often used with older build tooling, so this change would potentially break that.

This needs to be carefully analyzed to avoid fall outs similar to what we had to fix with https://github.com/dotnet/sdk/pull/14517.

[mkonijnenburg]

This solution does not work anymore with net sdk 6.0 (installed yesterday)

Directory.build.props:

<BaseOutputPath>bin\$(MSBuildProjectName)\</BaseOutputPath>
<MSBuildProjectExtensionsPath>obj\$(MSBuildProjectName)\</MSBuildProjectExtensionsPath>
<IntermediateOutputPath>obj\$(MSBuildProjectName)\$(Configuration)\</IntermediateOutputPath>

C:\Program Files\dotnet\sdk\6.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5): error NETSDK1004: Assets file 'D:\Applications\AAA\obj\BBB_yzakfkig_wpftmp\project.assets.json' not found. Run a NuGet package restore to generate this file. [D:\Applications\AAA\BBB_yzakfkig_wpftmp.csproj]

The error is only with WPF projects, the project.assests.json gets placed in the folder without the mangling (BBB).

[zivkan]

@mkonijnenburg since this is specific to WPF, it's related to WPF's targets/props (SDK). I created an issue there: https://github.com/dotnet/wpf/issues/5679

For what it's worth, the problem you found is unrelated to this issue. This isssue is about two csproj files in the same directory (or otherwise using the same obj folder). Your MSBuildProjectExtensionsPath contain the project name, hence every project has a different directory, they're not sharing the same directory, so there isn't a problem with two different projects trying to use the same project.assets.json file.

[mkonijnenburg]

Thanks for your reply. As extra context for the situation, I noticed this in multiple solutions where we use the Directory.build.props in a folder with two .csproj files. One for WPF development and the other for WinForms, sharing 90% of the files. When compiling without the props the nuget gets messed up. Using the props file resolves that, until yesterday with net SDK 6.0.

From: Andy Zivkovic @.> Sent: Wednesday, November 10, 2021 15:59 To: NuGet/Home @.> Cc: Marco Konijnenburg @.>; Mention @.> Subject: Re: [NuGet/Home] project.assets.json does not work with shared intermediates (#4463)

@mkonijnenburghttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmkonijnenburg&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542725684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iLe43uED7%2BzdE4SyFn1FSHxRoTwjOIgxyjJrpVilCCQ%3D&reserved=0 since this is specific to WPF, it's related to WPF's targets/props (SDK). I created an issue there: dotnet/wpf#5679https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fwpf%2Fissues%2F5679&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542725684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Lr4%2FoMSyX15KUiE%2FU9tIroLfAqKalFSZd6P9RnfW6qE%3D&reserved=0

For what it's worth, the problem you found is unrelated to this issue. This isssue is about two csproj files in the same directory (or otherwise using the same obj folder). Your MSBuildProjectExtensionsPath contain the project name, hence every project has a different directory, they're not sharing the same directory, so there isn't a problem with two different projects trying to use the same project.assets.json file.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNuGet%2FHome%2Fissues%2F4463%23issuecomment-965334485&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542735639%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qvXdfyfzs4tD1y%2BkGnnsHmvlj0rCYk3tT06Hi13YnxU%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABKURRIQQHXVIPAOIR3MPJDULKCD3ANCNFSM4C6O7BPQ&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542735639%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=txF%2FEegr%2FfUkIv4DZAdsKeFtMoGnf09wjw7m6BihoOI%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542745594%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yQaeNUzUZ8Dw0JJ4pibk%2F7csrPN74s1caj4UR0xNagY%3D&reserved=0 or Androidhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C2fd6e48a9ec8474752b208d9a45aa74f%7Cd1598a4048ac4fedb8b93dcb440ac6fa%7C0%7C0%7C637721531542745594%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZDHIceekPHic4ZWvFe65OQcOgmXYWMajXGOMRZrMurY%3D&reserved=0.

[Nirmal4G]

I hit this so many times but there's no way to rename project.assets.json file with say, MSBuildProjectName to actually keep those files separate. I mean, the props/targets and dgspec files have project name to disambiguate.

From what I saw in both NuGet and .NET SDK targets, the work needs to be in both. I can think of two solutions here.

Most projects don't need disambiguation as they are restored in to their own project folder but cases like the above happen when the same project folder contains two projects (may be targeting different versions of the dependencies). This could be fixed with #5154 as you then no longer need two project workarounds.

Another requirement to have many projects within the same folder is to simply build different assemblies with varying public surface API (think, Lite and Full modules). With .NET Framework, we have netmodule, in essence, Both VB and C# code can present in the same folder and each project produces a netmodule and links them together to create a single assembly (this is one of the most underrated features of .NET, I don't know why the team removed it in Core.).

With those reasons stated above, I can think of two solutions here. One is to enable #5154 and other is to have a disambiguation between restore assets by project name or file when they are in the same folder. The former ask could be bigger than just a simple refactor but the latter could be just that depending on how the assets file feature is implemented in the codebase.

I'm willing to take this on If the team accepts the request and guides me on where I should make the changes.

[BryanAldrich]

I found this and whilst exploring the code and various properties, I found that adding a Directory.build.props file with the following content resolved the build issue where the project.assets.json conflicts with two projects.

1) the project name the assets.json was built from should be in the file to detect multiple projects building into the same asset file. The rest of the output files seems to include the projectname, then file content, then extension. Seems that is reasonable. 2) Nuget should respect the IntermediateOutputPath variable. It does only currently honor the RestoreOutputPath or MSBuildProjectExtensionsPath. The former being defaulted to the latter. With at least 3 variables used for the 'obj' folder, not all of them being used equivalently either.

Just ideas and notes I had whilst resolving my current issue.

<Project>
  <!-- See https://aka.ms/dotnet/msbuild/customize for more details on customizing your build -->
  <PropertyGroup>
    <BaseIntermediateOutputPath>obj\$(Configuration)\$(MSBuildProjectName)\</BaseIntermediateOutputPath>
  </PropertyGroup>
</Project>

yes, i realize that i end up with $(Configuration) in my folder twice, but this lets me build debug and release versions of our internal nuget packages and swap them out on the fly. If IntermediateOutputPath was respected by nuget, then i could remove one layer, it's inconsequential as long it builds, deploys and runs.

[BryanAldrich]

To reply to my prior comment, it gets worse. The Azure DevOps task for Nuget Restore generates a new targets file for msbuild, then calls it. Thus, completely obliterating any chance of finding the Directory.Build.props file.

[nkolev92]

@BrightLight

You're hitting a limitation in NuGet.exe, tracked in https://github.com/NuGet/Home/issues/6734.

If possible, we recommend using msbuild /t:restore instead of the Nuget restore task. It even supports packages.config. https://learn.microsoft.com/en-us/nuget/reference/msbuild-targets#restoring-packagereference-and-packagesconfig-projects-with-msbuild.

Regarding the IntermediateOutputPath idea, NuGet is explicitly configuration agnostic, but that itself really related to this particular issue. MSBuildProjectExtensionsPath was added explicitly for NuGet, so if you want 2 projects in the same folder to not share the same intermediates, just make that's set to different paths for different projects.

[BryanAldrich]

@nkolev92 Thanks, I actually switched to the msbuild command yesterday, i'm glad though to see that it is a recommended path. Thanks for the advise and everything you do!

[Cygon]

I'm running into this with nearly any project I have, since I like to have my unit tests close to the code, i.e.

Nuclex.Support/
    Source/
        UsefulThing.cs

    Tests/
        UsefulThingTest.cs

    Nuclex.Support.csproj
    Nuclex.Support.Tests.csproj

Those .csproj files would be two .NET Standard 2.0 projects. I can set different intermediate directories (i.e. obj/source/... and obj/tests/...) and both assemblies will compile without issue.

Except when I introduce NuGet, i.e. to add NUnit or Moq as dependencies: the unit test project will fail to compile as if NUnit or Moq weren't present in the dependencies. There is no build message of any kind to indicate that the NUnit reference was invalid or dropped or anything - so it likely uses the dependencies of the non-unit-test project.

Since this project.assets.json is a mere intermediate file, couldn't it just be given a unique name or put in the user-defined intermediate directory?

[nkolev92]

You can use MSBuildProjectExtensionsPath to define a unique location for the assets file.