umbraco folder missing when running dotnet publish

[SteveVaneeckhout]

Which exact Umbraco version are you using? For example: 8.13.1 - don't just write v8

9.0.0-rc004

Bug summary

When running the command dotnet publish --output PublishDir the umbraco folder in the root of PublishDir is missing. The deployed website will not have a working backoffice.

Specifics

I'm using .NET Core 5 - No beta's of .NET Core 6 or VS installed.

Steps to reproduce

Open a commandline and run

mkdir umbraco9test
cd umbraco9test
dotnet new umbraco
dotnet publish --output publish

Notice the publish directory doesn't contain the umbraco folder.

Expected result / actual result

The umbraco folder should be there.

I asked this question on the forum and Søren responded with: run dotnet build before publish. This works on my machine but not in my Azure DevOps pipeline where I get the error:

error NETSDK1022: Duplicate 'Content' items were included. The .NET SDK includes 'Content' items from your project directory by default.
You can either remove these items from your project file, or set the 'EnableDefaultContentItems' property to 'false' if you want to explicitly include them in your project file.
For more information, see https://aka.ms/sdkimplicititems.

The duplicate items were:
'umbraco\PartialViewMacros\Templates\Breadcrumb.cshtml';
'umbraco\PartialViewMacros\Templates\EditProfile.cshtml';
'umbraco\PartialViewMacros\Templates\Empty.cshtml';
'umbraco\PartialViewMacros\Templates\Gallery.cshtml';
'umbraco\PartialViewMacros\Templates\ListAncestorsFromCurrentPage.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesFromChangeableSource.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesFromCurrentPage.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesOrderedByDate.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesOrderedByName.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesOrderedByProperty.cshtml';
'umbraco\PartialViewMacros\Templates\ListChildPagesWithDoctype.cshtml';
'umbraco\PartialViewMacros\Templates\ListDescendantsFromCurrentPage.cshtml';
'umbraco\PartialViewMacros\Templates\ListImagesFromMediaFolder.cshtml';
'umbraco\PartialViewMacros\Templates\Login.cshtml';
'umbraco\PartialViewMacros\Templates\LoginStatus.cshtml';
'umbraco\PartialViewMacros\Templates\MultinodeTree-picker.cshtml';
'umbraco\PartialViewMacros\Templates\Navigation.cshtml';
'umbraco\PartialViewMacros\Templates\RegisterMember.cshtml';
'umbraco\PartialViewMacros\Templates\SiteMap.cshtml';
'umbraco\UmbracoBackOffice\AuthorizeUpgrade.cshtml';
'umbraco\UmbracoBackOffice\Default.cshtml';
'umbraco\UmbracoBackOffice\Preview.cshtml';
'umbraco\UmbracoInstall\Index.cshtml';
'umbraco\UmbracoWebsite\NoNodes.cshtml';
'umbraco\config\appsettings-schema.json'

This might be a different issue that can be solved by adding <EnableDefaultContentItems>false</EnableDefaultContentItems> here: https://github.com/umbraco/Umbraco-CMS/blob/5bfab13dc5a268714aad2426a2b68ab5561a6407/build/NuSpecs/build/Umbraco.Cms.StaticAssets.targets#L3

[bergmania]

You will need to do a build explictly before you publish.

mkdir umbraco9test
cd umbraco9test
dotnet new umbraco
dotnet build
dotnet publish --output publish

[SteveVaneeckhout]

That works locally but not in my pipeline. The second command (dotnet publish) throws error NETSDK1022. This is the YAML File:

pool:
  vmImage: ubuntu-latest

variables:
  buildConfiguration: 'Release'
  SolutionPath: 'UmbLinTest.sln'

steps:
- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
    projects: '$(SolutionPath)'
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '$(SolutionPath)'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'

[mistyn8]

Thoughts from our journey.. azure devops and github actions

dotnet build (implicitly restores) msdocs and dotnet publish msdoc (implicitly restores and build) though we've found you do have to do the separate steps..

but presumably you should explicitly stop restore and build when adopting explicit steps otherwise you are building and restoring multiple times?

dotnet build --no-restore and dotnet publish --no-build (no-build also sets no-restore on publish)

so you would think that a dotnet publish would be enough as it should be restoring and building.. however looking at the logs it appears that the msbuild tasks for contentfile copy occur too late so files don't make it into the publish folder/package before publishing occurs so it's required to have individual steps.

also if you include an explicit restore step you can look for a cached nuget download to further optimise deployment :-)

[mistyn8]

also another gotcha... dotnet publish -c Release -o ./publish/ note the trailing slash for consistency if you want to reuse as a variable later on in msbuildtargets for instance.. $(PublishDir) <Copy SourceFiles="$(Projectdir)..\ExternalDependencies\Our.Umbraco.DocTypeGridEditor.dll" DestinationFiles="$(PublishDir)Our.Umbraco.DocTypeGridEditor.dll" />

also set your publish folder outside of the project root or set to ignore!

If not specified, it defaults to [project_file_folder]/bin/[configuration]/[framework]/publish/ for a framework-dependent executable and cross-platform binaries. It defaults to [project_file_folder]/bin/[configuration]/[framework]/[runtime]/publish/ for a self-contained executable.

In a web project, if the output folder is in the project folder, successive dotnet publish commands result in nested output folders. For example, if the project folder is myproject, and the publish output folder is myproject/publish, and you run dotnet publish twice, the second run puts content files such as .config and .json files in myproject/publish/publish.

[p-m-j]

Could you share the contents of your csproj, it should look like this https://github.com/umbraco/Umbraco-CMS/blob/v9/dev/build/templates/UmbracoProject/UmbracoProject.csproj which has sensible defaults for the umbraco folder

<None Include="umbraco\**\*.*"> <!-- <<< this (from forum) shouldn't be required -->

[mistyn8]

Sure Also this has grown with us from beta003 up to rc002 so you might have some things in there to accommodate wwwroot for instance ending up in the published structure.. (app_plugins/views also wasn't in the early beta if memory serves)

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="8.1.1" />
    <PackageReference Include="Bergmania.OpenStreetMap" Version="1.0.0" />
    <PackageReference Include="Our.Umbraco.DocTypeGridEditor" Version="9.0.0-beta001" />
    <PackageReference Include="RestSharp" Version="106.12.0" />
    <PackageReference Include="Serilog.Sinks.Seq" Version="5.0.1" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.1.5" />
    <PackageReference Include="Umbraco.Cms" Version="9.0.0-rc002" />
    <PackageReference Include="Umbraco.Cms.Web.Common" Version="9.0.0-rc002" />
    <PackageReference Include="Umbraco.Forms" Version="9.0.0-beta003" />
    <PackageReference Include="uSync" Version="9.0.0-rc002" />
  </ItemGroup>

  <ItemGroup>
    <Compile Remove="umbraco\Data\**" />
    <Compile Remove="umbraco\logs\**" />
    <Compile Remove="umbraco\MediaCache\**" />
  </ItemGroup>
  <ItemGroup>
    <EmbeddedResource Remove="umbraco\Data\**" />
    <EmbeddedResource Remove="umbraco\logs\**" />
    <EmbeddedResource Remove="umbraco\MediaCache\**" />
  </ItemGroup>
  <ItemGroup>
    <None Remove="umbraco\Data\**" />
    <None Remove="umbraco\logs\**" />
    <None Remove="umbraco\MediaCache\**" />
    <None Include="config\**\*.*">
      <ExcludeFromSingleFile>true</ExcludeFromSingleFile>
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
    </None>
    <None Include="umbraco\**\*.*">
      <ExcludeFromSingleFile>true</ExcludeFromSingleFile>
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
    <Content Remove="umbraco\Data\**" />
    <Content Remove="umbraco\logs\**" />
    <Content Remove="umbraco\MediaCache\**" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\AAA.Core.ModelsBuilder\TSD.Core.ModelsBuilder.csproj" />
    <ProjectReference Include="..\AAA.Core\TSD.Core.csproj" />
  </ItemGroup>
  <ItemGroup>
    <Content Include="App_Plugins\**">
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
    </Content>
    <Content Include="Views\**\*.cshtml">
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
    </Content>
    <Content Include="appsettings.json" CopyToOutputDirectory="Always" />
    <Content Include="appsettings.Development.json" DependentUpon="appsettings.json" CopyToOutputDirectory="Always" />
    <Content Include="appsettings.Production.json" DependentUpon="appsettings.json" CopyToOutputDirectory="Always" />
    <Content Include="wwwroot\**">
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
    </Content>
    <Content Include="uSync\v9\**" CopyToOutputDirectory="Always" />
  </ItemGroup>
  <PropertyGroup>
    <EnableDefaultContentItems>false</EnableDefaultContentItems>
  </PropertyGroup>
  <!-- stops dotnet watch baulking on model generation -->
  <ItemGroup>
    <Compile Remove="umbraco\models\**" />
    <Compile Include="umbraco\models\**" Watch="false" />
  </ItemGroup>  
  <!-- Set this to true if ModelsBuilder mode is not InMemoryAuto-->
  <PropertyGroup>
    <RazorCompileOnBuild>false</RazorCompileOnBuild>
    <RazorCompileOnPublish>false</RazorCompileOnPublish>
    <UserSecretsId>7e012c32-e8d7-46c6-8cb3-f571b91f5a48</UserSecretsId>
  </PropertyGroup>
</Project>

I'd agree with the forum that shouldn't be required as restoring from nuget when deploying. But left in for the moment.

[p-m-j]

So I guess this should get you most of the way

<Project Sdk="Microsoft.NET.Sdk.Web">

    <PropertyGroup>
        <TargetFramework>net5.0</TargetFramework>
        <DefaultItemExcludes>$(DefaultItemExcludes);App_Plugins/**;</DefaultItemExcludes>
        <DefaultItemExcludes>$(DefaultItemExcludes);umbraco/**;</DefaultItemExcludes>
        <DefaultItemExcludes>$(DefaultItemExcludes);wwwroot/media/**;</DefaultItemExcludes>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="8.1.1" />
      <PackageReference Include="Bergmania.OpenStreetMap" Version="1.0.0" />
      <PackageReference Include="Our.Umbraco.DocTypeGridEditor" Version="9.0.0-beta001" />
      <PackageReference Include="RestSharp" Version="106.12.0" />
      <PackageReference Include="Serilog.Sinks.Seq" Version="5.0.1" />
      <PackageReference Include="Swashbuckle.AspNetCore" Version="6.1.5" />
      <PackageReference Include="Umbraco.Cms" Version="9.0.0-rc002" />
      <PackageReference Include="Umbraco.Cms.Web.Common" Version="9.0.0-rc002" />
      <PackageReference Include="Umbraco.Forms" Version="9.0.0-beta003" />
      <PackageReference Include="uSync" Version="9.0.0-rc002" />
    </ItemGroup>

    <ItemGroup>
        <Content Include="App_Plugins/**" CopyToOutputDirectory="Always" />
        <Content Include="umbraco/**" CopyToOutputDirectory="Always" />
        <Content Remove="umbraco/Data/**" />
        <Content Remove="umbraco/Logs/**" />
        <Content Remove="umbraco/mediacache/**" />
        <Content Remove="umbraco/models/**" />
        <Compile Include="umbraco/models/**" Exclude="**/*.flag" />
    </ItemGroup> 

     <ItemGroup>
      <ProjectReference Include="..\AAA.Core.ModelsBuilder\TSD.Core.ModelsBuilder.csproj" />
      <ProjectReference Include="..\AAA.Core\TSD.Core.csproj" />
    </ItemGroup>

    <ItemGroup>
       <Content Include="uSync\v9\**" CopyToOutputDirectory="Always" />
    </ItemGroup>

    <!-- Set this to true if ModelsBuilder mode is not InMemoryAuto-->
    <PropertyGroup>
        <RazorCompileOnBuild>false</RazorCompileOnBuild>
        <RazorCompileOnPublish>false</RazorCompileOnPublish>
        <UserSecretsId>7e012c32-e8d7-46c6-8cb3-f571b91f5a48</UserSecretsId>
    </PropertyGroup>

</Project>

[mistyn8]

@p-m-j Thanks I'll try with a clean v9 released site ;-) (so everything back to standard)

Did you have any thoughts on the implicit restore/build and removing them if you were performing those individual steps explicitly?

Or should a dotnet publish now work as a single step, with the above csproj? (as it implicitly restores and builds)

[p-m-j]

There's a PR which I believe resolves the issue where build is required before publish, https://github.com/umbraco/Umbraco-CMS/pull/11211 but for now there's no harm in running build first.

$ dotnet build 
$ dotnet publish --no-build # avoid duplicate work

[SteveVaneeckhout]

It might be related to this issue: https://github.com/dotnet/sdk/issues/14622

[p-m-j]

Cheers but pretty sure this one should have been closed a while back.