nx-dotnet / nx-dotnet

A Nx plugin adding support for .NET 5+ (or .net core) projects, featuring full project graph and generator support.
https://www.nx-dotnet.com/
MIT License
255 stars 52 forks source link

[BUG] Dependencies not automatically resolved during build #820

Open jvskriubakken opened 6 months ago

jvskriubakken commented 6 months ago

Current Behavior

Hi,

I keep on experiencing, during build that referenced assembly cannot be found. The csproj of the app has a ProjectReference to the project.

The workaround is that I have to add the referenced assembly as an implicit dependency to the app's project.json.

What could cause this? Could there any fragility in nx-dotnet's resolving of dependencies? As I understand it nx-dotnet should resolve probject dependencies automatically by inspecting csproj files?

Expected Behavior

That nx-dotnet automatically resolves dependencies from csproj-files.

Github Repo

No response

Steps to Reproduce

1.

Nx Report

Node   : 18.17.1
   OS     : darwin-arm64
   yarn   : 1.22.19

   nx                 : 17.2.8
   @nx/js             : 17.2.8
   @nx/jest           : 17.2.8
   @nx/linter         : 17.2.8
   @nx/eslint         : 17.2.8
   @nx/workspace      : 17.2.8
   @nx/cypress        : 17.2.8
   @nx/devkit         : 17.2.8
   @nx/eslint-plugin  : 17.2.8
   @nx/plugin         : 17.2.8
   @nx/react          : 17.2.8
   @nx/storybook      : 17.2.8
   @nrwl/tao          : 17.2.8
   @nx/web            : 17.2.8
   @nx/webpack        : 17.2.8
   typescript         : 5.2.2
   ---------------------------------------
   Community plugins:
   @nx-dotnet/core : 2.1.2
   ---------------------------------------
   Local workspace plugins:
         @airmont/nx-plugins

✨  Done in 1.66s.

nx.json

{
  "plugins": ["@nx-dotnet/core"],
  "namedInputs": {
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "production": [
      "default",
      "!{projectRoot}/.eslintrc.json",
      "!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
      "!{projectRoot}/tsconfig.spec.json",
      "!{projectRoot}/jest.config.[jt]s",
      "!{projectRoot}/.storybook/**/*",
      "!{projectRoot}/**/*.stories.@(js|jsx|ts|tsx|mdx)",
      "!{projectRoot}/src/test-setup.[jt]s",
      "!{projectRoot}/tsconfig.storybook.json"
    ],
    "sharedGlobals": ["{workspaceRoot}/babel.config.json"]
  },
  "targetDefaults": {
    "build": {
      "dependsOn": [
        "^build",
        "codegen",
        "^codegen",
        "codegen",
        "^codegen",
        "codegen",
        "^codegen"
      ],
      "inputs": ["production", "^production"],
      "cache": true
    },
    "e2e": {
      "inputs": ["default", "^production"],
      "cache": true
    },
    "lint": {
      "inputs": [
        "default",
        "{workspaceRoot}/.eslintrc.json",
        "{workspaceRoot}/.eslintignore"
      ],
      "cache": true
    },
    "test": {
      "inputs": ["default", "^production", "{workspaceRoot}/jest.preset.js"],
      "cache": true
    },
    "build-storybook": {
      "inputs": [
        "default",
        "^production",
        "{projectRoot}/.storybook/**/*",
        "{projectRoot}/tsconfig.storybook.json"
      ],
      "cache": true
    },
    "@nx/jest:jest": {
      "inputs": ["default", "^production", "{workspaceRoot}/jest.preset.js"],
      "cache": true,
      "options": {
        "passWithNoTests": true
      },
      "configurations": {
        "ci": {
          "ci": true,
          "codeCoverage": true
        }
      }
    }
  },
  "generators": {
    "@nx/react": {
      "application": {
        "style": "styled-components",
        "linter": "eslint",
        "bundler": "webpack",
        "babel": true
      },
      "component": {
        "style": "styled-components"
      },
      "library": {
        "style": "styled-components",
        "linter": "eslint",
        "unitTestRunner": "jest"
      }
    }
  },
  "$schema": "./node_modules/nx/schemas/nx-schema.json"
}

Failure Logs

> nx run firefly-my-chimney-webapi-cs:build:production

Executing Command: dotnet "--version"
8.0.101
Executing Command: dotnet "build" "C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj" "--configuration" "Release" "--no-dependencies" "false" "--verbosity" "minimal"
MSBuild version 17.8.3+195e7f5a3 for .NET
  Determining projects to restore...
C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Storage.Timeseries\Firefly.Storage.Timeseries.csproj : warning NU1902: Package 'System.Data.SqlClient' 4.8.4 has a known moderate severity vulnerability, https://github.com/advisories/GHSA-8g2p-5pqh-5jmc [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Storage.Timeseries\Firefly.Storage.Timeseries.csproj : warning NU1903: Package 'System.Data.SqlClient' 4.8.4 has a known high severity vulnerability, https://github.com/advisories/GHSA-98g6-xh36-x2p7 [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Provisioning.Services\Firefly.Provisioning.Services.csproj : warning NU1903: Package 'Azure.Identity' 1.7.0 has a known high severity vulnerability, https://github.com/advisories/GHSA-5mfx-4wcx-rv27 [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Storage.KeyVault\Firefly.Storage.KeyVault.csproj : warning NU1903: Package 'Azure.Identity' 1.7.0 has a known high severity vulnerability, https://github.com/advisories/GHSA-5mfx-4wcx-rv27 [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
  Restored C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Provisioning.Data\Firefly.Provisioning.Data.csproj (in 451 ms).
  Restored C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Storage.KeyVault\Firefly.Storage.KeyVault.csproj (in 451 ms).
  Restored C:\BuildAgent\_work\10\s\libs\firefly\my-chimney\cs\service\service.csproj (in 531 ms).
  Restored C:\BuildAgent\_work\10\s\libs\firefly\shared\cs\Firefly.Shared.Service\Firefly.Shared.Service.csproj (in 530 ms).
  Restored C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Core.Application.Web\Firefly.Core.Application.Web.csproj (in 531 ms).
  Restored C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Core.Web\Firefly.Core.Web.csproj (in 468 ms).
  Restored C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Provisioning.Services\Firefly.Provisioning.Services.csproj (in 531 ms).
  Restored C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj (in 535 ms).
  9 of 17 projects are up-to-date for restore.
C:\Program Files\dotnet\sdk\8.0.101\Sdks\Microsoft.NET.Sdk.Razor\targets\Microsoft.NET.Sdk.Razor.MvcApplicationPartsDiscovery.targets(54,5): warning RAZORSDK1007: Reference assembly C:\BuildAgent\_work\10\s\dist\libs\Firefly\shared\cs\Firefly.Core.Application.Web\net6.0\Firefly.Core.Application.Web.dll could not be found. This is typically caused by build errors in referenced projects. [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
CSC : error CS0006: Metadata file 'C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Core.Application.Web\obj\Release\net6.0\ref\Firefly.Core.Application.Web.dll' could not be found [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]
CSC : error CS0006: Metadata file 'C:\BuildAgent\_work\10\s\libs\Firefly\shared\cs\Firefly.Storage.KeyVault\obj\Release\net6.0\ref\Firefly.Storage.KeyVault.dll' could not be found [C:\BuildAgent\_work\10\s\apps\firefly\my-chimney\webapi-cs\Airmont.Firefly.MyChimney.WebApiCs.csproj]

Build FAILED.

Additional Information

csproj of the app (firefly-my-chimney-webapi-cs) that fails during build:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>7813e2b2-f546-44d9-bf13-ca8b6ecf45df</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.2.3" />
    <PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="6.0.7" />
    <PackageReference Include="Serilog.AspNetCore" Version="6.0.1" />
    <PackageReference Include="Serilog.Enrichers.Environment" Version="2.2.0" />
    <PackageReference Include="Serilog.Sinks.ApplicationInsights" Version="4.0.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\service\service.csproj" />
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\Ef\Ef.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Core.Application.Web\Firefly.Core.Application.Web.csproj" />
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\domain\domain.csproj"/>
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Provisioning.Services\Firefly.Provisioning.Services.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Storage.DocumentDb\Firefly.Storage.DocumentDb.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Storage.KeyVault\Firefly.Storage.KeyVault.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Content Remove="project.json" />
    <None Include="project.json">
      <CopyToOutputDirectory>Never</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>
AgentEnder commented 6 months ago

I've not experienced this, is it possible that you are running commands from other directories than the workspace root? That shouldn't matter, but would be nice to rule out.

jvskriubakken commented 6 months ago

All commands are run from workspace/repo root.

jvskriubakken commented 6 months ago

webapp project.json:

{
  "name": "firefly-my-chimney-webapi-cs",
  "$schema": "../../../../node_modules/nx/schemas/project-schema.json",
  "projectType": "application",
  "sourceRoot": "apps/firefly/my-chimney/webapi-cs",
  "implicitDependencies": ["firefly-my-chimney-client-ts"],
  "targets": {
    "build": {
      "executor": "@nx-dotnet/core:build",
      "outputs": ["{options.output}"],
      "options": {
        "configuration": "Debug",
        "noDependencies": false
      },
      "configurations": {
        "production": {
          "configuration": "Release"
        }
      }
    },
    "serve": {
      "executor": "@nx-dotnet/core:serve",
      "options": {
        "configuration": "Debug"
      },
      "configurations": {
        "production": {
          "configuration": "Release"
        }
      }
    },
    "serve-production": {
      "executor": "@nx-dotnet/core:serve",
      "options": {
        "configuration": "Debug",
        "launch-profile": "Production"
      },
      "configurations": {
        "production": {
          "configuration": "Release"
        }
      }
    },
    "lint": {
      "executor": "@nx-dotnet/core:format"
    },
    "format": {
      "executor": "@nx-dotnet/core:format",
      "options": {
        "fix": true
      }
    },
    "publish": {
      "executor": "@nx-dotnet/core:publish",
      "configurations": {
        "production": {
          "configuration": "Release"
        }
      }
    },
    "publish-to-pipeline": {
      "executor": "@airmont/nx-plugins:publish-pipeline-artifact",
      "options": {
        "dotNetVersion": "net8.0"
      }
    }
  },
  "tags": ["dotnet"]
}

is it correct to have "noDependencies": false here?

webapp csproj:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>7813e2b2-f546-44d9-bf13-ca8b6ecf45df</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.2.3" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="6.0.7" />
    <PackageReference Include="Serilog.AspNetCore" Version="6.0.1" />
    <PackageReference Include="Serilog.Enrichers.Environment" Version="2.2.0" />
    <PackageReference Include="Serilog.Sinks.ApplicationInsights" Version="4.0.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\service\service.csproj" />
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\Ef\Ef.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Core.Application.Web\Firefly.Core.Application.Web.csproj" />
    <ProjectReference Include="..\..\..\..\libs\firefly\my-chimney\cs\domain\domain.csproj"/>
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Provisioning.Services\Firefly.Provisioning.Services.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Storage.DocumentDb\Firefly.Storage.DocumentDb.csproj" />
    <ProjectReference Include="..\..\..\..\libs\Firefly\shared\cs\Firefly.Storage.KeyVault\Firefly.Storage.KeyVault.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Content Remove="project.json" />
    <None Include="project.json">
      <CopyToOutputDirectory>Never</CopyToOutputDirectory>
    </None>
  </ItemGroup>
</Project>

... here you can see that the dependency Firefly.Core.Application.Web.csproj is listed.

Here you have firefly-shared-cs-core-web-application:

{
  "name": "firefly-shared-cs-core-web-application",
  "$schema": "../../../../../node_modules/nx/schemas/project-schema.json",
  "projectType": "library",
  "sourceRoot": "libs/firefly/shared/cs/Firefly.Core.Application.Web",
  "targets": {
    "build": {
      "executor": "@nx-dotnet/core:build",
      "outputs": ["{options.output}"],
      "options": {
        "configuration": "Debug",
        "noDependencies": true
      },
      "configurations": {
        "production": {
          "configuration": "Release"
        }
      }
    },
    "test": {
      "executor": "@nx-dotnet/core:test",
      "options": {}
    },
    "lint": {
      "executor": "@nx-dotnet/core:format"
    },
    "format": {
      "executor": "@nx-dotnet/core:format",
      "options": {
        "fix": true
      }
    }
  },
  "tags": ["dotnet"]
}

and it's csproj :

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\Firefly.Core.Security\Firefly.Core.Security.csproj" />
    <ProjectReference Include="..\Firefly.Core.Storage\Firefly.Core.Storage.csproj" />
    <ProjectReference Include="..\Firefly.Core.Web\Firefly.Core.Web.csproj" />
    <ProjectReference Include="..\Firefly.Shared.Service\Firefly.Shared.Service.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Compile Remove="bin\**" />
  </ItemGroup>

  <ItemGroup>
    <EmbeddedResource Remove="bin\**" />
  </ItemGroup>

  <ItemGroup>
    <None Remove="bin\**" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.21.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.21.0" />
    <PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="6.0.7" />
    <PackageReference Include="Serilog" Version="2.11.0" />
    <PackageReference Include="Serilog.Sinks.ApplicationInsights" Version="4.0.0" />
  </ItemGroup>

</Project>

... are they as expected? Thinking especiallhy of noDependencies": true?

Tantol commented 4 months ago

Same issue. In my opinion it's something wrong with "noDependencies".

When I run nx build <project_name> with below project.json configuration:

  "targets": {
    "build": {
      "options": {
        "noDependencies": false
      }
    }
  }

Everything works as indented. This runs: dotnet "build" "/home/x/y/z.csproj" "--configuration" "Debug" "--verbosity" "minimal"

But when I run same command with this configuration (default one):

  "targets": {
    "build": {
      "options": {
        "noDependencies": true
      }
    }
  }

I get error CSC : error CS0006: Metadata file: This runs: dotnet "build" "/home/x/y/z.csproj" "--configuration" "Debug" "--no-dependencies" "--verbosity" "minimal".

There is something wrong with: https://www.nx-dotnet.com/docs/core/guides/incremental-builds "(...) This is not the case within Nx, due to the targetDependencies setup inside nx.json."

Are we missing some configuration in nx.json for dotnet?

Tantol commented 4 months ago

I found issue. MSBuild properties are not supported for nx.

Example: If in .csproj i use relative path:

(...) 
<ProjectReference Include="..\..\..\..\libs\x\x.csproj" />

Build with "--no-dependencies" works and nx graph show everything as expected.

But when i use MSBuild properties like this: Directory.Build.props:

    <RepoRoot>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)'))</RepoRoot>

.csproj:

(...) 
<ProjectReference Include="$(RepoRoot)\libs\x\x.csproj" />

nx graph show no connection with my x library and build with "--no-dependencies" throw CSC : error CS0006: Metadata file

mxa0079 commented 3 months ago

I have a similar issue, only happening in my CI pipeline, and the workaround is to disable NX cache.

Also, FWIW this problem started to happen after I added a new .net project to my repo. Before I already had a set of projects and everything was working fine.

UPDATE:

The comments in this thread helped. My problem was resolved by editing project.json and changing the "noDependencies" option to false. However, it seems like in the new project this value was set to true by default. Is this expected? What is the purpose of this option?

"options": {
        "configuration": "Debug",
        "noDependencies": false
      }

HOWEVER, we had other branches with this new project (even main) that are successfully building without the aforementioned option. So there seems to be some inconsistent behavior either with NX or with the .NET plugin for NX and it seems to be related to this option and the use of cache.

In summary:

Why did this problem not happen before? Why did it appear in the following branch and not on the original PR? Why disabling cache fixed the issue?

So many questions.

I hope this helps the NX team to figure out what is going on.

Other than these issues that pop from time to time (especially with the .NET plugin), we love using NX!

AgentEnder commented 3 months ago

The reasoning for us setting noDependencies is described here: https://www.nx-dotnet.com/docs/core/guides/incremental-builds.

The fact that skipping the cache results in things working suggests that Nx isn't capturing all of the build outputs that it should. By default, nx-dotnet sets up outputs going into {workspaceRoot}/dist/{projectRoot} and {workspaceRoot}/dist/intermediates/{projectRoot}. If there are outputs from your project not in those directories, they wouldn't be captured by Nx and thus not replayed from the cache.

Missing intermediates causes issues when noDependencies is true.

BryceBarbara commented 4 weeks ago

I also ran into this issue and it proved quite difficult to debug. The issue is that the cloud cache was messed up (it was also missing the ref .dll) and we couldn't figure out how to clear the cache stored there. We ended up basically just using NX as we would lerna.

We wanted to use NX and its features but the reality is that it's not worth the extra headache when also working with .NET projects.