app/appsettings.Development.json: path not found

[Danielku15]

Strangely for me the happy-path of publishing a container according to the example tutorial already fails for me. I remember that with older .net SDKs and NuGet Packages it used to work already on my machine. Considering that I'm the first reporting this I would guess it is not some general problem. What could cause such an error?

Steps to reproduce: I followed the exact steps described in https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?view=vs-2022

1. dotnet new worker -o Worker -n DotNet.ContainerImage
2. cd worker
3. dotnet run -> Works
4. dotnet add package Microsoft.NET.Build.Containers
5. dotnet publish --os linux --arch x64 /t:PublishContainer -c Release -> Fails

Error output

C:\Users\daniel\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : Failed to load image to local Docker daemon. stdout: re-exec error: exit status 1: stderr: app/appsettings.Development.json: path not found [D:\Dev\Other\DockerPublish\Worker\DotNet.ContainerImage.csproj]
C:\Users\daniel\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :  [D:\Dev\daniel\DockerPublish\Worker\DotNet.ContainerImage.csproj]

Environment Windows 11 Enterprise 21H2 Docker Version: Docker version 20.10.22, build 3a2c30b Docker Desktop Version: 4.16.2 DotNet Version: 7.0.102 NuGet Package Version: 0.3.2

Further details If I execute dotnet publish /t:PublishContainer -c Release I get a slightly different error with windows paths, but the error is similar:

C:\Users\daniel\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : Failed to load image to local Docker daemon. stdout: re-exec error: exit status 1: output: open \\?\C:\Program Data\Docker\tmp\hcs1287968575\app\appsettings.Development.json: The system cannot find the path specified. [D:\Dev\Other\DockerPublish\Worker\DotNet.ContainerImage.csproj]
C:\Users\daniel\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :  [D:\Dev\Other\DockerPublish\Worker\DotNet.ContainerImage.csproj]

MSBuild BinLog: msbuild.binlog.zip Project Zip: Worker.zip

[baronfel]

Can you take and provide an https://aka.ms/binlog? It provides a lot more detail about your build environment that might help diagnose this.

[Danielku15]

Uploaded a binlog to the initial post.

[baronfel]

In that binlog, the RuntimeIdentifier property didn't seem to get set. Can you confirm that your publish command either had the --arch and --os options or used the --runtime option? The Command line arguments are reported by MSBuild don't have any of that information, like I'd expect.

As a result, the tooling used the RID of your local SDK (win-x64) to pick a container, but a Working Directory that wasn't Windows-friendly (/app), and I think that's causing the actual issue here.

[Danielku15]

My bad 🙇 I updated the binlog. Command used: dotnet publish --os linux --arch x64 /t:PublishContainer -c Release -bl. I also uploaded my project directory to be sure to have everything together.

[baronfel]

@Danielku15 is your local Docker install setup to use Windows Containers or Linux Containers? Only one can be configured at a time, and the only docs I can find for the error message in your most recent binlog suggest Windows Containers are in use.

[baronfel]

You should be able to gather information about your local Docker install via docker info - I used docker info --format '{{json .}}' to get JSON, which looks like this for me:

{
  "ID": "<READACTED>",
  "Containers": 11,
  "ContainersRunning": 2,
  "ContainersPaused": 0,
  "ContainersStopped": 9,
  "Images": 313,
  "Driver": "overlay2",
  "DriverStatus": [
    [
      "Backing Filesystem",
      "extfs"
    ],
    [
      "Supports d_type",
      "true"
    ],
    [
      "Native Overlay Diff",
      "true"
    ],
    [
      "userxattr",
      "false"
    ]
  ],
  "Plugins": {
    "Volume": [
      "local"
    ],
    "Network": [
      "bridge",
      "host",
      "ipvlan",
      "macvlan",
      "null",
      "overlay"
    ],
    "Authorization": null,
    "Log": [
      "awslogs",
      "fluentd",
      "gcplogs",
      "gelf",
      "journald",
      "json-file",
      "local",
      "logentries",
      "splunk",
      "syslog"
    ]
  },
  "MemoryLimit": true,
  "SwapLimit": true,
  "KernelMemory": true,
  "KernelMemoryTCP": true,
  "CpuCfsPeriod": true,
  "CpuCfsQuota": true,
  "CPUShares": true,
  "CPUSet": true,
  "PidsLimit": true,
  "IPv4Forwarding": true,
  "BridgeNfIptables": true,
  "BridgeNfIp6tables": true,
  "Debug": false,
  "NFd": 75,
  "OomKillDisable": true,
  "NGoroutines": 78,
  "SystemTime": "2023-01-24T20:40:41.984447291Z",
  "LoggingDriver": "json-file",
  "CgroupDriver": "cgroupfs",
  "CgroupVersion": "1",
  "NEventsListener": 8,
  "KernelVersion": "5.10.102.1-microsoft-standard-WSL2",
  "OperatingSystem": "Docker Desktop",
  "OSVersion": "",
  "OSType": "linux",
  "Architecture": "x86_64",
  "IndexServerAddress": "https://index.docker.io/v1/",
  "RegistryConfig": {
    "AllowNondistributableArtifactsCIDRs": [],
    "AllowNondistributableArtifactsHostnames": [],
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ],
    "IndexConfigs": {
      "docker.io": {
        "Name": "docker.io",
        "Mirrors": [],
        "Secure": true,
        "Official": true
      },
      "hubproxy.docker.internal:5000": {
        "Name": "hubproxy.docker.internal:5000",
        "Mirrors": [],
        "Secure": false,
        "Official": false
      }
    },
    "Mirrors": []
  },
  "NCPU": 16,
  "MemTotal": 16738717696,
  "GenericResources": null,
  "DockerRootDir": "/var/lib/docker",
  "HttpProxy": "http.docker.internal:3128",
  "HttpsProxy": "http.docker.internal:3128",
  "NoProxy": "hubproxy.docker.internal",
  "Name": "docker-desktop",
  "Labels": [],
  "ExperimentalBuild": false,
  "ServerVersion": "20.10.21",
  "Runtimes": {
    "io.containerd.runc.v2": {
      "path": "runc"
    },
    "io.containerd.runtime.v1.linux": {
      "path": "runc"
    },
    "runc": {
      "path": "runc"
    }
  },
  "DefaultRuntime": "runc",
  "Swarm": {
    "NodeID": "",
    "NodeAddr": "",
    "LocalNodeState": "inactive",
    "ControlAvailable": false,
    "Error": "",
    "RemoteManagers": null
  },
  "LiveRestoreEnabled": false,
  "Isolation": "",
  "InitBinary": "docker-init",
  "ContainerdCommit": {
    "ID": "770bd0108c32f3fb5c73ae1264f7e503fe7b2661",
    "Expected": "770bd0108c32f3fb5c73ae1264f7e503fe7b2661"
  },
  "RuncCommit": {
    "ID": "v1.1.4-0-g5fd4c4d",
    "Expected": "v1.1.4-0-g5fd4c4d"
  },
  "InitCommit": {
    "ID": "de40ad0",
    "Expected": "de40ad0"
  },
  "SecurityOptions": [
    "name=seccomp,profile=default"
  ],
  "Warnings": [
    "WARNING: No blkio throttle.read_bps_device support",
    "WARNING: No blkio throttle.write_bps_device support",
    "WARNING: No blkio throttle.read_iops_device support",
    "WARNING: No blkio throttle.write_iops_device support"
  ],
  "ClientInfo": {
    "Debug": false,
    "Context": "default",
    "Plugins": [
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Docker Inc.",
        "Version": "v0.9.1",
        "ShortDescription": "Docker Buildx",
        "Name": "buildx",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-buildx.exe"
      },
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Docker Inc.",
        "Version": "v2.13.0",
        "ShortDescription": "Docker Compose",
        "Name": "compose",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-compose.exe"
      },
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Docker Inc.",
        "Version": "v0.0.5",
        "ShortDescription": "Docker Dev Environments",
        "Name": "dev",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-dev.exe"
      },
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Docker Inc.",
        "Version": "v0.2.16",
        "ShortDescription": "Manages Docker extensions",
        "Name": "extension",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-extension.exe"
      },
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Anchore Inc.",
        "Version": "0.6.0",
        "ShortDescription": "View the packaged-based Software Bill Of Materials (SBOM) for an image",
        "URL": "https://github.com/docker/sbom-cli-plugin",
        "Name": "sbom",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-sbom.exe"
      },
      {
        "SchemaVersion": "0.1.0",
        "Vendor": "Docker Inc.",
        "Version": "v0.22.0",
        "ShortDescription": "Docker Scan",
        "Name": "scan",
        "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-scan.exe"
      }
    ],
    "Warnings": null
  }
}

The key field is OSType, which for me is Linux.

Potentially a work item here would be to inspect the local docker configuration via docker info --format '{{ json. }}' and verify that the OSType field is compatible with the RID of the image we're about to load.

[Danielku15]

Your assumption is partially true: Yes I am having Docker with Windows Containers configured. BUT 😁: I have also enabled the experimental flag in the docker config which allows me to build and run Windows and Linux container successfully.

My Docker Info:

{
    "ID": "######",
    "Containers": 0,
    "ContainersRunning": 0,
    "ContainersPaused": 0,
    "ContainersStopped": 0,
    "Images": 15,
    "Driver": "windowsfilter (windows) lcow (linux)",
    "DriverStatus": [["Windows", ""], ["LCOW", ""]],
    "Plugins": {
        "Volume": ["local"],
        "Network": ["ics", "internal", "l2bridge", "l2tunnel", "nat", "null", "overlay", "private", "transparent"],
        "Authorization": null,
        "Log": ["awslogs", "etwlogs", "fluentd", "gcplogs", "gelf", "json-file", "local", "logentries", "splunk", "syslog"]
    },
    "MemoryLimit": false,
    "SwapLimit": false,
    "KernelMemory": false,
    "KernelMemoryTCP": false,
    "CpuCfsPeriod": false,
    "CpuCfsQuota": false,
    "CPUShares": false,
    "CPUSet": false,
    "PidsLimit": false,
    "IPv4Forwarding": true,
    "BridgeNfIptables": true,
    "BridgeNfIp6tables": true,
    "Debug": false,
    "NFd": -1,
    "OomKillDisable": false,
    "NGoroutines": 33,
    "SystemTime": "2023-01-25T09:14:22.4827047+01:00",
    "LoggingDriver": "json-file",
    "CgroupDriver": "",
    "NEventsListener": 4,
    "KernelVersion": "10.0 22000 (22000.1.amd64fre.co_release.210604-1628)",
    "OperatingSystem": "Windows 10 Enterprise Version 2009 (OS Build 22000.1455)",
    "OSVersion": "10.0.22000",
    "OSType": "windows",
    "Architecture": "x86_64",
    "IndexServerAddress": "https://index.docker.io/v1/",
    "RegistryConfig": {
        "AllowNondistributableArtifactsCIDRs": [],
        "AllowNondistributableArtifactsHostnames": [],
        "InsecureRegistryCIDRs": ["127.0.0.0/8"],
        "IndexConfigs": {
            "docker.io": {
                "Name": "docker.io",
                "Mirrors": [],
                "Secure": true,
                "Official": true
            }
        },
        "Mirrors": []
    },
    "NCPU": 16,
    "MemTotal": 34006982656,
    "GenericResources": null,
    "DockerRootDir": "C:\\ProgramData\\Docker",
    "HttpProxy": "",
    "HttpsProxy": "",
    "NoProxy": "",
    "Name": "#############",
    "Labels": [],
    "ExperimentalBuild": true,
    "ServerVersion": "20.10.22",
    "Runtimes": null,
    "DefaultRuntime": "",
    "Swarm": {
        "NodeID": "",
        "NodeAddr": "",
        "LocalNodeState": "inactive",
        "ControlAvailable": false,
        "Error": "",
        "RemoteManagers": null
    },
    "LiveRestoreEnabled": false,
    "Isolation": "hyperv",
    "InitBinary": "",
    "ContainerdCommit": {
        "ID": "",
        "Expected": ""
    },
    "RuncCommit": {
        "ID": "",
        "Expected": ""
    },
    "InitCommit": {
        "ID": "",
        "Expected": ""
    },
    "SecurityOptions": null,
    "ProductLicense": "Community Engine",
    "Warnings": null,
    "ClientInfo": {
        "Debug": false,
        "Context": "default",
        "Plugins": [{
                "SchemaVersion": "0.1.0",
                "Vendor": "Docker Inc.",
                "Version": "v0.10.0",
                "ShortDescription": "Docker Buildx",
                "Name": "buildx",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-buildx.exe"
            }, {
                "SchemaVersion": "0.1.0",
                "Vendor": "Docker Inc.",
                "Version": "v2.15.1",
                "ShortDescription": "Docker Compose",
                "Name": "compose",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-compose.exe"
            }, {
                "SchemaVersion": "0.1.0",
                "Vendor": "Docker Inc.",
                "Version": "v0.0.5",
                "ShortDescription": "Docker Dev Environments",
                "Name": "dev",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-dev.exe"
            }, {
                "SchemaVersion": "0.1.0",
                "Vendor": "Docker Inc.",
                "Version": "v0.2.17",
                "ShortDescription": "Manages Docker extensions",
                "Name": "extension",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-extension.exe"
            }, {
                "SchemaVersion": "0.1.0",
                "Vendor": "Anchore Inc.",
                "Version": "0.6.0",
                "ShortDescription": "View the packaged-based Software Bill Of Materials (SBOM) for an image",
                "URL": "https://github.com/docker/sbom-cli-plugin",
                "Name": "sbom",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-sbom.exe"
            }, {
                "SchemaVersion": "0.1.0",
                "Vendor": "Docker Inc.",
                "Version": "v0.23.0",
                "ShortDescription": "Docker Scan",
                "Name": "scan",
                "Path": "C:\\Program Files\\Docker\\cli-plugins\\docker-scan.exe"
            }
        ],
        "Warnings": null
    }
}

deamon.json

{
  "experimental": true,
  "hosts": [
    "npipe:////./pipe/docker_engine_windows"
  ]
}

I switched to linux containers and then I could build the image successful. Also as a follow up I checked what happens if I try to publish a Windows container --os win with Windows containers active. mcr.microsoft.com seems currently a bit overloaded and unresponsive (heard about Company internal rumours that overall Azure suffering outages right now 😱 ) to me so it took me some attempts to build it. But it also fails with a weird issue:

dotnet publish --os win --arch x64 /t:PublishContainer -c Release -bl leads to

C:\Users\kusc\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9):
 error : Failed to load image to local Docker daemon. stdout: re-exec error: exit status 1: output: open \\?\C:\Program
Data\Docker\tmp\hcs3424113302\app\appsettings.Development.json: The system cannot find the path specified. [D:\Dev\Othe
r\DockerPublish2\Worker\DotNet.ContainerImage.csproj]
C:\Users\kusc\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9):
 error :  [D:\Dev\Other\DockerPublish2\Worker\DotNet.ContainerImage.csproj]

msbuild.binlog.zip

[baronfel]

I'm not sure that we will be able to easily support this mode of operation. The epic tracking support for this lcow mode is here, and docker save/docker load don't even appear to be started yet.

I think the last error you have shown (the app\appsettings.Development.json: The system cannot find the path specified one) is coming from an incorrect default ContainerWorkingDirectory. The default for this sets the value to a linux-rooted path - can you try setting this property manually to a path you know to exist in your windows container? If you do so, then the Entrypoint should be automatically corrected.

[Danielku15]

I added <ContainerWorkingDirectory>C:\app\</ContainerWorkingDirectory> and looking at the resulting path it seems to generate a wrong path Likely something is wrong here in this area?

D:\Dev\Other\DockerPublish2\Worker>dotnet publish --os win --arch x64 /t:PublishContainer -c Release -bl
MSBuild version 17.4.1+9a89d02ff for .NET
C:\Program Files\dotnet\sdk\7.0.102\MSBuild.dll --property:_IsPublishing=true -property:Configuration=Release -property:RuntimeIdentifier=win-x64 -property:SelfContained=false -bl -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\7.0.102\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,C:\Program Files\dotnet\sdk\7.0.102\dotnet.dll -maxcpucount -restore -target:Publish /t:PublishContainer -verbosity:m .\DotNet.ContainerImage.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  DotNet.ContainerImage -> D:\Dev\Other\DockerPublish2\Worker\bin\Release\net7.0\win-x64\DotNet.ContainerImage.dll
  DotNet.ContainerImage -> D:\Dev\Other\DockerPublish2\Worker\bin\Release\net7.0\win-x64\publish\
  Building image 'dotnet-worker-image' with tags 1.0.0 on top of base image mcr.microsoft.com/dotnet/runtime:7.0
C:\Users\kusc\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9):
 error : Failed to load image to local Docker daemon. stdout: re-exec error: exit status 1: output: open \\?\C:\Program
Data\Docker\tmp\hcs4883562\C:\app\appsettings.Development.json: path contains invalid character `:` [D:\Dev\Other\Docke
rPublish2\Worker\DotNet.ContainerImage.csproj]
C:\Users\kusc\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9):
 error :  [D:\Dev\Other\DockerPublish2\Worker\DotNet.ContainerImage.csproj]

I tried following values:

• C:\app
• C:\app\
• C:/app
• C:/app/
• app
• app\

all with similar errors that the file is not found.

I will try to setup a local dev environment for this repo and see if I can narrow down some things.

[Danielku15]

I dived now into some details and have some findings which might be relevant. I haven't reworked yet the things to match the Image of Docker but will do that soon so see if this would fix the problems.

Project Setup:

1. I started with the same project as in the initial post and added<ContainerWorkingDirectory>C:\app</ContainerWorkingDirectory>.
2. I extended sdk-container-builds with a new option which will write a PAX tarball file (not GNU like in this repo) to disk for comparison. I changed to PAX because docker save also uses Posix format. I wanted to have the same structure visible in 7zip.
3. I created a fresh .net console application in Rider with enabled windows docker support (see attachment) and then built with docker build + save to tar via docker save.
4. Then I compared the tarballs according to their content.

Findings:

• I noticed that both with PAX and GNU tarballs 7zip is complaining about "Unexpected end of data" 😱
• I noticed quite a discrepancy in the working dirs and entry points of the two images.
• The layer holding the application is quite different between the docker built image and the ones from sdk-container-builds.
  • docker build has only a simple tar while sdk-container-builds seem to have a tar.gz resulting in an additional nesting.
  • docker build has both a "Files" and "Hives" (Registry) subfolder, while sdk-container-builds has directly the files inside the tar.gz.
  • docker build has only a folder "app" in the files followed by the files while the sdk-container builds has a "C:" folder and then a "app" folder.

Questions:

a) Should this project maybe better write POSIX tarballs like docker? It is likely not significant but would be closer to what docker is doing. b) Should we correct the JSON generation to use dotnet as entry point and ensure the correct path formatting? c) Should the code building the layer tarball be better changed to output the same structure like docker?

• Simple Tar instead of a tar.gz
• Skip drive letter folder in the tar
• Put the files into a Files subfolder d) Has anyone yet noticed this error on the tar file where 7zip claims it is incomplete?

NetDockerWin.zip

[baronfel]

Great analysis - I'll take this point by point and try to explain what I think isn't directly impactful vs what is a relevant difference we should investigate:

• PAX vs GNU format - we have https://github.com/dotnet/sdk-container-builds/issues/159 for making this change, we're not married to GNU format
• Entrypoint changes - the docker-generated manifest is using the dotnet host as the application launcher, while we chose the 'native' apphost generated by MSBuild. There is no functional difference between them, and in fact if you change your application to not be self-contained then our tooling will also use the dotnet host as the app launcher.
• Tar layout differences - I would want to see some documentation on expected layer tarball formats for windows before making any changes. It feels to me that we're doing the right thing for Linux, but there are some behavioral differences for Windows that we need to take into account. Windows-specific changes are tracked and worked in #117

So overall, I would say that overall our behavior for Windows containers needs additional work, but Windows containers are not a primary focus for us so I'm not sure yet when that would be scheduled.

[Danielku15]

I have a small update on the topic: I re-exectued my tests from above but this time with Linux containers. And I likely found the reason why even Linux containers do not work.

My test steps:

1. I switched to Linux containers
2. I created again an image with a Dockerfile and one with sdk-container-builds and dumped them to local tarballs.
3. I changed to Windows containers.
4. I changed sdk-container-builds to write tarballs to disk using PAX instead of Gnu and direct docker load commands
5. I checked the resulting tarball for differences against the sdk-container-builds. I also tried docker load on the tarball to see if it can be loaded. The tarball from the dockerfile works, the output of sdk-container-builds fails.
6. I noticed that the tarball is still claimed to be incomplete and noticed one bug. this line is lacking a using statement which leads to incomplete data. After this fix docker load still fails
7. I manually uncompressed the tarball of sdk-container-builds and then also the layer.tar of my application
8. I manually recompressed the layer.tar and the image.tar
9. Executed docker load and get an error with the layer hash.
10. Fix the sha256 hash of the last layer in the {image-hash}.json
11. Compress the image.tar again
12. docker load Works!

Finding: There is a problem in the writing of the layer.tar. When only repackaging the image.tar docker still refuses to load. Looking at the tar -ztvf of the two layer.tar there is a significant difference. The one which is working has a directory entry app while the one not working has only the files!

Conclusion:

1. We need to properly dispose the TarWriter here
2. We need to ensure we create all directory records here

I have no full fix ready yet but will try to prepare one soon where I will traverse the file system tree and create the correct TarEntry instances.

[baronfel]

Going to close this because it was fixed in #323