Failed to upload schema2 manifest: manifest invalid: manifest invalid

[Jaspreetc]

Description

Failed to upload schema2 manifest: manifest invalid: manifest invalid

Briefly describe the problem you are having in a few paragraphs.

I am trying to push Windows Container image to Jfrog Artifactory Docker Registry it fails with an error “failed to upload schema2 manifest “

Similar thing happens when I try to push an image to Docker private Registry it fails with an error “manifest blob unknown: blob unknown to registry”

Steps to reproduce the issue:

C:\Windows\system32>docker push xx/windowservercore
The push refers to a repository [xx]
cc6b0a07c696: Skipped foreign layer
3fd27ecef6a3: Skipped foreign layer
manifest invalid: manifest invalid

Note: Using xx to hide the internal repo information

docker push xx.xx.xx.xxx:5000/windowsservercore
The push refers to a repository [xx.xx.xx.xx:5000/windowsservercore]
cc6b0a07c696: Skipped foreign layer
3fd27ecef6a3: Skipped foreign layer```
errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

Describe the results you received:

Log Name:      Application
Source:        docker
Date:         
10/14/2016 6:51:45 AM
Event ID:      1
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      WIN-H3VK04MET11
Description:
failed to upload schema2 manifest: manifest invalid: manifest invalid

Log Name:      Application
Source:        docker
Date:          10/14/2016
6:51:45 AM
Event ID:      1
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WIN-H3VK04MET11
Description:
Not continuing with push
after error: manifest invalid: manifest invalid

Log Name:      Application
Source:        docker
Date:          10/14/2016 6:53:48 AM
Event ID:      1
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WIN-H3VK04MET11
Description:
Not continuing with push after error: errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

Log Name:      Application
Source:        docker
Date:          10/14/2016 6:53:48 AM
Event ID:      1
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      WIN-H3VK04MET11
Description:
failed to upload schema2 manifest: errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

Describe the results you expected: Successful upload the image to repo

Additional information you deem important (e.g. issue happens only occasionally): Dockerd 1.12 & 1.12.2 also have this issue.

Output of docker version:

C:\Windows\system32>docker -v
Docker version 1.13.0-dev, build 535f52c
C:\Windows\system32>docker -v
Docker version 1.13.0-dev, build 535f52c (paste your output here)

More information on Docker Image 

C:\Windows\system32>docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
microsoft/windowsservercore        latest              93a9c37b36d0        3 weeks ago         8.68 GB

Output of docker info:

C:\Windows\system32> docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.13.0-dev
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: nat null overlay
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.0.amd64fre.rs1_release.160715-1616)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 5
Total Memory: 3 GiB
Name: WIN-H3VK04MET11
ID: QJYG:RN3K:GOEM:XST5:QMSJ:XRJA:NBFN:OPE3:A7ZS:XPPI:QQZQ:V6KB
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 xxx.xxx.xxx.xxx:5000
 127.0.0.0/8
Live Restore Enabled: false

C:\Windows\system32>

Additional environment details (AWS, VirtualBox, physical, etc.): Artifactory Enterprise : 4.9.0 rev 40226 Docker Registry version: Docker-registry-0.9.1-7.el7.x86_64

I will be opening a support ticket with Jfrog Artifactory as well, but wanted to check with the newer version of Dockerd engine will the schema version will always be on version 2 or will there be a fallback to version 1 or if this is a bug in docker ?

[mabrarov]

To @Jaspreetc:

I will be opening a support ticket with Jfrog Artifactory as well

Hi. Was you able to open JFrog Artifactory ticket? I'd like to know ticket ID and link.

[Jaspreetc]

To @mabrarov the ticket has been opened but the logs are yet to be updated which may not be before November 7th . in the mean time can you please let me know if docker 1.13 dev has a fall back mechanism to older schema ? I will ping you with the details by November 7 as soon as the logs has been uploaded.

[mabrarov]

To @Jaspreetc:

can you please let me know if docker 1.13 dev has a fall back mechanism to older schema

Could you share details (steps, what needs to be done, what logs need to be collected) with me (refer to email in my GitHub profile), please? I'm using JFrog Arifactory docker image - web UI of Artifactory reports about 4.14.0 version. I'm also using Windows Server 2016 RTM Standard (the docker host from which I try to push my docker images based on official microsoft/windowsservercore image) with docker (output of docker version) 1.12.2-cs2-ws-beta.

[cpuguy83]

Docker-registry-0.9.1-7.el7.x86_64 -- this registry is not compatible with schema2 being used by docker. You need to update your registry version. Not registry v2 is here: https://github.com/docker/distribution

[Jaspreetc]

Thanks @cpuguy83 (Brian) let me give that a try and will get back to you shortly. I am off for a week so it won't be before Nov 10

[Jaspreetc]

@mabrarov As soon as I collect the logs with the Jfrog team will let you know more information about it. It will be prolly post my return

[mabunixda]

@mabrarov At the moment we run into the same problem. I raised a bug on their jira

[thaJeztah]

Let me close this issue here, because this is not a bug in docker, but feel free to continue the discussion.

[Jaspreetc]

@cpuguy83 Hi Brian, I have tried the latest version but it still fails with the same error Name : docker-distribution Version : 2.4.1 Release : 2.el7 Architecture: x86_64 Install Date: Thu 18 Aug 2016 11:32:08 AM EDT Group : Unspecified Size : 15789976 License : ASL 2.0 Signature : RSA/SHA256, Mon 25 Jul 2016 02:46:56 AM EDT, Key ID 199e2f91fd431d51 Source RPM : docker-distribution-2.4.1-2.el7.src.rpm Build Date : Sat 25 Jun 2016 05:57:02 PM EDT Build Host : x86-039.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. http://bugzilla.redhat.com/bugzilla Vendor : Red Hat, Inc. URL : https://github.com/docker/distribution Summary : Docker toolset to pack, ship, store, and deliver content Description : Docker toolset to pack, ship, store, and deliver content

Logs from Docker-Registry Nov 17 08:05:07 registry: time="2016-11-17T08:05:07-05:00" level=info msg="response completed" go.version=go1.6.2 http.request.host="xxx.xxx.xxx.xxx:5000" http.request.id=cb7c52e3-3021-4415-83f7-0f9319e1afee http.request.method=GET http.request.remoteaddr="xxx.xxx.xxx.xxx:35483" http.request.uri="/v2/" http.request.useragent="docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.603878ms http.response.status=200 http.response.written=2 instance.id=b6fe19c1-9422-4f65-bc6a-31bf633d9c00 version="v2.4.0+unknown" Nov 17 08:05:07 registry: xxx.xxx.xxx.xxx - - [17/Nov/2016:08:05:07 -0500] "GET /v2/ HTTP/1.1" 200 2 "" "docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" Nov 17 08:05:07 registry: time="2016-11-17T08:05:07-05:00" level=info msg="response completed" go.version=go1.6.2 http.request.host="xxx.xxx.xxx.xxx:5000" http.request.id=55452b5a-2182-4ed1-bfa4-19ccd1d185ec http.request.method=HEAD http.request.remoteaddr="xxx.xxx.xxx.xxx:35483" http.request.uri="/v2/windowsservercore/blobs/sha256:f49a4ea104f107e5c35d9ab75ca079b3fd4b256c5743794cc15434fd3b4c38b9" http.request.useragent="docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" http.response.contenttype="application/octet-stream" http.response.duration=3.089281ms http.response.status=200 http.response.written=0 instance.id=b6fe19c1-9422-4f65-bc6a-31bf633d9c00 version="v2.4.0+unknown" Nov 17 08:05:07 registry: xxx.xxx.xxx.xxx - - [17/Nov/2016:08:05:07 -0500] "HEAD /v2/windowsservercore/blobs/sha256:f49a4ea104f107e5c35d9ab75ca079b3fd4b256c5743794cc15434fd3b4c38b9 HTTP/1.1" 200 0 "" "docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" Nov 17 08:05:07 registry: time="2016-11-17T08:05:07-05:00" level=error msg="response completed with error" err.code="manifest blob unknown" err.detail=sha256:9c7f9c7d9bc2915388ecc5d08e89a7583658285469d7325281f95d8ee279cc60 err.message="blob unknown to registry" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="xxx.xxx.xxx.xxx:5000" http.request.id=010ab6f8-bdb9-4167-8f34-36edbb21eb07 http.request.method=PUT http.request.remoteaddr="xxx.xxx.xxx.xxx:35483" http.request.uri="/v2/windowsservercore/manifests/latest" http.request.useragent="docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.836276ms http.response.status=400 http.response.written=319 instance.id=b6fe19c1-9422-4f65-bc6a-31bf633d9c00 vars.name=windowsservercore vars.reference=latest version="v2.4.0+unknown" Nov 17 08:05:07 registry: time="2016-11-17T08:05:07-05:00" level=error msg="response completed with error" err.code="manifest blob unknown" err.detail=sha256:d33fff6043a134da85e10360f9932543f1dfc0c3a22e1edd062aa9b088a86c5b err.message="blob unknown to registry" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="xxx.xxx.xxx.xxx:5000" http.request.id=010ab6f8-bdb9-4167-8f34-36edbb21eb07 http.request.method=PUT http.request.remoteaddr="xxx.xxx.xxx.xxx:35483" http.request.uri="/v2/windowsservercore/manifests/latest" http.request.useragent="docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.148474ms http.response.status=400 http.response.written=319 instance.id=b6fe19c1-9422-4f65-bc6a-31bf633d9c00 vars.name=windowsservercore vars.reference=latest version="v2.4.0+unknown" Nov 17 08:05:07 registry: xxx.xxx.xxx.xxx - - [17/Nov/2016:08:05:07 -0500] "PUT /v2/windowsservercore/manifests/latest HTTP/1.1" 400 319 "" "docker/1.13.0-dev go/go1.7.1 git-commit/535f52c os/windows arch/amd64 UpstreamClient(Docker-Client/1.13.0-dev (windows))"

[Jaspreetc]

Jfrog accepted the problem and have update it might get fixed in Q2 2017 https://www.jfrog.com/jira/browse/RTFACT-10305

[Jaspreetc]

@thaJeztah , I believe its a bug or design change required for Docker, since its not working in Docker Registry Version 2.4.1 as well. If there is a working solution available to push Windows Container images to Private Docker Registry please share, else there is some work required on this to get it fixed.

[thaJeztah]

@Jaspreetc looks like that's not an official version of the registry, but a version packaged by Red Hat. Support for foreign layers was added in version 2.5 https://github.com/docker/distribution/releases/tag/v2.5.0, and is needed for Windows based images because the base layers are only allowed to be distributed by Microsoft (for legal reasons). When pushing an image based on those images, only your layers will be pushed to the registry, but the base layer is not pushed (a reference to the foreign location is included in the image's manifest).

To run the current version of the registry, you can use the official image; https://github.com/docker/docker.github.io/blob/master/registry/deploying.md

[Jaspreetc]

@thaJeztah Thanks it was downloaded from https://github.com/docker/distribution. Let me try the newer version and get back to you. Thanks for your valuable input appreciate your quick response.

[Jaspreetc]

@thaJeztah I deployed the registry "https://github.com/docker/distribution/releases/tag/v2.5.0,",yesterday but it fails to run I get the following error , either there is an issue with the way I have deployed registry or there is something wrong with the image, I have tried it on 2 different machine and have the same result.

[root@~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry v2.5 1d5334ee7f4b 20 hours ago 289.2 MB golang 1.7-alpine 4e874ba52406 4 weeks ago 241.1 MB [root@~]# docker run -it -p 5000:5000 --restart=always --name re gistry 1d5334ee7f4b docker: Error response from daemon: No command specified. See 'docker run --help'.

[thaJeztah]

@Jaspreetc did you build that image yourself or pull it from Docker hub? The image you showed is 289MB, but the official image is only 10MB (compressed, uncompressed +/- 33MB)

What happens if you simply run the official image from Docker Hub?

docker pull registry:2.5
docker run -it -p 5000:5000 --restart=always --name registry registry:2.5

[Jaspreetc]

@thaJeztah I downloaded the source code and build the container image, the same thing was done for 2.4.1 and it was working.

Let me try the steps you have requested and would get back to you on Monday. if that does not work I will try 2.5.1 or 2.6.0 Rc1 as well.

[thaJeztah]

Building from GitHub works for me as well (although you'd get a much larger image, as it's meant for development, not for "production", so not a minimal image);

docker build -t registry:v2.5.0 github.com/docker/distribution#v2.5.0

Sending build context to Docker daemon 18.92 MB
Step 1 : FROM golang:1.6-alpine
1.6-alpine: Pulling from library/golang
...
...
Successfully built bea1acc7f1ea

docker run -it --rm -p 5000:5000 --name registry registry:v2.5.0

INFO[0000] debug server listening localhost:5001
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  environment=development go.version=go1.6.3 instance.id=5f0ea88c-15df-4b94-8d04-243f9a6458f4 service=registry version=v2.5.0

(same for v2.5.1)

[Jaspreetc]

@thaJeztah Thanks that worked, but I am curious as to why do I need to get onto internet every time if I have to download the Windows container image, even when I have downloaded it once into my environment, doesn't it defeat the purpose? why can't I reference it to the downloaded image , since not all servers would have access to internet. If this would be the case then it would be difficult to pull the image on new servers and that was the whole reason for registry to be in place correct?

[thaJeztah]

@Jaspreetc How exactly do you mean "why do I need to get onto internet every time if I have to download the Windows container image"? If you docker pull <some-windows-image> and then docker run <some-windows-image> does it try to pull the image each time?

[Jaspreetc]

@thaJeztah

Server 1 – Windows Container image downloaded from Docker hub Server 2 – Image Pushed to Docker registry , but only the latest digest goes on to the registry and the foreign layers are skipped

C:\Windows\system32>docker push xxx.xxx.xxx.xxx:5000/windowsservercore The push refers to a repository [xxx.xxx.xxx.xxx:5000/windowsservercore] b9454c3094c6: Skipped foreign layer 3fd27ecef6a3: Skipped foreign layer latest: digest: sha256:571381e3e11fb93ba128adbdcaeb354cb6a470ee60919b89627731361b021d6e size: 943

Server3 Docker pull xxx.xxx.xxx.xxx:5000/windowsservercore ( docker registry)

time="2016-11-21T08:45:42.009251300-08:00" level=debug msg="pulling blob \"sha256:d33fff6043a134da85e10360f9932543f1dfc0c3a22e1edd062aa9b088a86c5b\"" time="2016-11-21T08:45:42.009251300-08:00" level=debug msg="pulling blob \"sha256:9c7f9c7d9bc2915388ecc5d08e89a7583658285469d7325281f95d8ee279cc60\"" time="2016-11-21T08:45:42.010254400-08:00" level=debug msg="Pulling sha256:d33fff6043a134da85e10360f9932543f1dfc0c3a22e1edd062aa9b088a86c5b from foreign URL https://go.microsoft.com/fwlink/?linkid=834677" time="2016-11-21T08:45:42.012253300-08:00" level=debug msg="Pulling sha256:9c7f9c7d9bc2915388ecc5d08e89a7583658285469d7325281f95d8ee279cc60 from foreign URL https://go.microsoft.com/fwlink/?linkid=830340" time="2016-11-21T08:54:53.022940200-08:00" level=debug msg="Downloaded d33fff6043a1 to tempfile C:\Users\ADMINI~1\AppData\Local\Temp\2\GetImageBlob260816119"

It detects the foreign layers and pulls the blob & image from Microsoft site, it defeats the purpose of registry 

[thaJeztah]

Yes, the foreign layers will always be pulled from the Microsoft servers (however, once they're in the cache of that server, it will use the ones that are cached locally).

Unfortunately, that's for legal reasons; the base layers are not allowed to be distributed through other servers. I don't know if there's exemption from this, but I don't think there are.

[Jaspreetc]

@thaJeztah

doesn't it sounds obscure to download roughly 4-5 gig of image via internet on each and every Container host which may be in hundreds? Microsoft has to come up with something better alternatives cause this doesn't seems to be a viable solution for each pull from registry :( .. is there a way to raise a request to change this feature, at least where the image will be made available offline like they do on WSUS for patches which can then be locally pulled over intranet

[thaJeztah]

@Jaspreetc this was the best that could be realized; previously, the base layer had to be manually installed on each host. @jhowardmsft or @jstarks may know if future changes are still possible.

[lowenna]

@Jaspreetc this was the best that could be realized; previously, the base layer had to be manually installed on each host. @jhowardmsft or @jstarks may know if future changes are still possible.

Leaving this one for @patricklang. This isn't really a technical question requiring a code change to docker.

[friism]

@Jaspreetc you're ok to push the foreign layers to your own private registry, as long as you don't redistribute them publicly (this is the equivalent of storing a Windows ISO on a private server).

Unfortunately, the Docker distribution source code does not currently have a way to force-push foreign layers, but we're exploring workarounds.

An alternative approach (that I don't know whether works in your scenario) is to pre-load base layers to whatever server or OS image that you use when setting up new machines. That way, docker-enabled servers will have the relevant base layers when they're brought up.

cc @dmp42 @stevvooe

[Jaspreetc]

@jhowardmsft Thanks John @friism - the alternative approach will work only for the initial pull, but the cycle has to repetitive with Windows container image updated every month that makes thing trickier. A thought if we can explore onto modifying the part where it calls the foreign layer to be downloaded from Microsoft to local intranet; provided the blob/digest and SHA1 are intact even when the link location changes.

@dmp42 @stevvooe

[friism]

@Jaspreetc yeah, I also tried just modifying the JSON in the image tarball to remove the foreign layer references. The bits are there on disc so it seems like that should work, but I haven't been successful pushing foreign layers to private repo yet.

[friism]

Alright, here's what you can do as a workaround:

1. `docker save -o image.tar.gz
2. Open the tarball, grab manifest.json and remove the LayerSources stuff
3. Package tarball back up with the modified manifest.json
4. [annoying part begins here] Get a hold of a Docker engine that doesn't have the foreign layers. Approaches include:
   • stop Docker Engine and start again with -g set to non-standard location
   • have separate clean engine on different host
5. docker load -i image.tar.gz into that fresh Docker engine (eg. use -H to point at it)
6. docker tag <your-favorite-foreign-layer-image> <private-repo-tag>
7. docker push <private-repo-tag>

This is obviously not great, but should work as a workaround.

[Jaspreetc]

@friism Thanks Michael, let me give it a try for the kick start, I still believe there is a need to permanently fix this for ease of operation.

[Jaspreetc]

@friism That did work as a workaround. Thanks

[friism]

@thaJeztah can I suggest that this issue be re-opened? I think it's a missing feature that there's no way to force-push foreign layer to private registries.

[charlessolar]

I was chatting with @friism in the other bug - I don't know what to make of all these comments being not a docker dev but this is the situation I brought up to ask for a re-open.

I have a windows build server building my docker images. I then have a production environment of 4 machines which I want to run those images on.
In TP5 I was able to push images from the build server to a private repository and start these images in the production environment by pulling from that repository.

This process doesn't work anymore in GA and is a blocking issue for us. All I'm asking for is a way to get an image built on a foreign machine onto a production machine without untarring and modifying manifests as above. For right now the only way I know to do this is to transfer the complete docker build to each production server and do docker build [image] . 4 times.

[friism]

@volak out of curiosity, are you pushing to a private docker/distribution registry that you're running? Foreign layers should work with that, although they'll be situated on Docker Hub.

[charlessolar]

@friism Yes I am running registry:2 in my own environment. But I get the same issue as everyone else

[Jaspreetc]

I think foreign layers issue is fixed post 2.5 and it does not work in 2.0

[Jaspreetc]

Registry:2.5 or 2.5.1

[charlessolar]

Yep! Switching to registry:2.5.1 allows me to push windows images again! Theres still a small issue that the image has to be tagged with port :443 otherwise docker complains about unknown ssl authorities but I think thats something I should be able to figure out. Thanks

[Jaspreetc]

@friism Hey Michael, with this issue being reopen , when are we expecting the changes to be in effect to force the foreign layer images to be pushed to registry. I understand there might quite a code change required but wanted to know if there can be an ETA for this.

[arthi-ramachandran]

@friism Hi Michael, am trying a docker save -o c:\docker\images\.tar on windows. Apparely, the command sits there and does not return for a while. Is there a way to know whether the command is doing anything ?

[friism]

@arthi-ramachandran recommend opening task manager and sorting processes by disk i/o - it should be writing to disk

[kenchenaz]

@friism Hey, same issue here, is there a workaround for pushing images to private repo(Jfrog Artifactory) with a manifest being generated? I keep getting manifest invalid and then can't pull that image from my repo?

[Jaspreetc]

@kenchenaz There is an issue with Jfrog, please check with Jfrog, they were supposed to fix this by Dec 2016

[friism]

@jstarks with your experience from building the original foreign layer implementation, I wanted to ask if you had time to implement an override switch that would let users force push foreign layers to private registries?

cc @aaronlehmann @dmp42 @vsaraswat

[arthi-ramachandran]

@friism , that will be really useful

[stevvooe]

I wanted to ask if you had time to implement an override switch that would let users force push foreign layers to private registries?

If we add this flag, why have foreign layers at all?

[friism]

why have foreign layers

@stevvooe this is so that when Windows images are distributed publicly, the Windows base layers can be distributed by Microsoft. Once the flag is added, base layers are distributed by Microsoft for public images on Docker Hub and by default also for Windows images hosted in private repos. But for Windows images in private registries, there's an additional option to push and store the Windows base layers in the private registry.

[aaronlehmann]

If we add this flag, why have foreign layers at all?

There are some scenarios (such as air gaps) where it may not be possible or efficient to retrieve layers from central servers. I feel we should allow users to push foreign layers in these exceptional circumstances, rather than being coercive. What's most important with foreign layers is not to push them by default, to avoid unintentional copyright infringement. However, it should ultimately be the user's decision, with our guidance, on how to handle them.

[thecloudtaylor]

As per Windows licensing terms and the supplemental EULA for container images (https://docs.microsoft.com/en-us/virtualization/windowscontainers/images-eula and/or licence.txt file in each image) customers are permitted limited redistribution rights. Specifically, they can push base images to their own repositories for distribution to their own hosts. We specifically added this clause to address air-gap scenarios but we need to ensure that a user does not unknowingly push an image to a public repro thus publicly distributing Windows.

[jstarks]

@friism I'm supportive of such a change, with the caveat that the private registry needs to preserve the foreign layer metadata. We must ensure that a foreign layer pulled from a private registry cannot be pushed to a public registry (or other registry that doesn't accept foreign layers).

However, I don't think I'll have time to tackle this for a while. If someone else has time, I'm happy to review the changes.

[stevvooe]

I feel we should allow users to push foreign layers in these exceptional circumstances, rather than being coercive.

It seems that these circumstances are not exceptional. This sounds like the norm for working with Windows images on premises. We should make a regular user work flow that doesn't accuse the user of piracy or end up with a user just using a flag they don't understand. Both result in accidental misunderstanding that can affect workflow.

Can we figure out a way to make this work so that users don't blindly add this flag to every docker pull command because they only understand "I need this to make it work"?

However, it should ultimately be the user's decision, with our guidance, on how to handle them.

Agreed, but foreign layers were never meant to be a part of the UX. It opens up too many questions. These issues have already come up in other PRs and issues and there doesn't seem to be a plan to address them. Once you have a flag that says "go head and push foreign layers", at what point do you create a feature to craft them? When everyone is crafting these layers, how do you maintain the user experience?

We can completely avoid this issue by automating the behavior with private registries or possibly warning the user they are about to distribute something that may have restrictions.

we need to ensure that a user does not unknowingly push an image to a public repro thus publicly distributing Windows.

Let's make sure that is the narrow requirement. I think we can make a limited feature that solves this issue without introducing a flag or the concept of "foreign layers" to users.