search

vmware / vic

vSphere Integrated Containers Engine is a container runtime for vSphere.
http://vmware.github.io/vic
Other
640 stars 174 forks source link

Docker pull failing because of to many empty layer (a3ed95caeb02) pulled #8321

Open kimbamps opened 6 years ago

kimbamps commented 6 years ago

Summary

Hello, we are setting up a POC environment to see if VIC is a valid use case for us. But for now we are stuck with the fact that we cannot pull/run our own container images. Pulling images with a few layers works, but the more layer our images have, to more time the "empty image layer" is pulled and eventually the pull crashes with following error. All our images work on our docker swarm. This error is telling us that our containers contain more that 100 layers. but if I pull them on our docker swarm they only have between 10-30 layers

Failed to write to image store: [POST /storage/{store_name}][500] WriteImage default &{Code:500 Message:ServerFaultCode: XML element depth exceeds configured maximum 100

Environment information

For this POC we are using: Vsphere 6.0.0 Vic 1.4.0 Docker client 18.03.1-ce VCH without TLS

Steps to reproduce

Is this example you can see the behavior, but I took the smallest image so that It would not crash, but you can clearly see that the "empty" layer is extracted multiple times. I don't think this is normal...

$ docker -H <IP_VCH:2375> pull ubuntu:14.04
14.04: Pulling from library/ubuntu
72c01b436656: Pull complete
65584f5f70ee: Pull complete
dc9874b52952: Pull complete
86656bbaa6fd: Pull complete
7fe6916ab382: Pull complete
**a3ed95caeb02**: Pull complete => Empty Layer
Digest: sha256:5545d04a2ad5de4b5a13a97ce603539b24679292a897f5db36038879b139c05f
Status: Downloaded newer image for library/ubuntu:14.04

Here we see that the official ubuntu image is been pulled with one empty layer. This empty layer has apperantly something to do with the registry manifest version being used... see links lower.

If we pull this ubuntu-14.04 with some extra tools and settings, nothing fancy. We see that the empty layer is been pullen many times. Actually during the pull I see that the empty layer is only downloaded ones, but extracted multiple times

$ docker -H <IP_VCH:2375> pull <privateDockerHub>/linux:ubuntu-14.04
ubuntu-14.04: Pulling from <privateDockerHub>/linux
cf0a75889057: Pull complete
c8de9902faf0: Pull complete
a3c0f7711c5e: Pull complete
e6391432e12c: Pull complete
624ce029a17f: Pull complete
a3ed95caeb02: Pull complete => extracted multiple times
2149d461260c: Pull complete
6c283b9acc5e: Pull complete
e052bf1293d2: Pull complete
1710f0a06637: Pull complete
0b16be660659: Pull complete
Digest: sha256:0c2a0ba12cb5bd285cb12abc4a6ce905a8271cc1f94fd7b24d33ad229f7a1a9a
Status: Downloaded newer image for <privateDockerHub>/linux:ubuntu-14.04

When I try to delete the images you can clearly see how many times this empty layer was extracted. (in bold)

$ docker -H <IP_VCH:2375> images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              14.04               c32fae490809        5 weeks ago         188MB
<privateDockerHub>/linux         ubuntu-14.04        d99e689b7dc7        17 months ago       288MB
$ docker -H <IP_VCH:2375> rmi  d99e689b7dc7 c32fae490809
Deleted: 0b16be6606597fd437f2e328a804dc6d53d313f39c3e77677d7e03ef6c102691
Deleted: 1710f0a06637f4a85e53833c82fab6bc1c2a2e459a941c739432473c67daacd9
Deleted: e052bf1293d2d9e431338cd9c07134367fe5bcf38f699fdec9fb58cd8da7b215
Deleted: 6c283b9acc5e0bc748bda42e9796278daffc9f407969db2efb0f99aa2660dd3a
Deleted: 2149d461260c6c995bfff971173fc6ce56d3a257172684320be65418fabdeabd
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  => deleted multiple times
Deleted: 624ce029a17f47b8a009a436b1190e11820261f63408a1e7e213d30845c1d0ab
Deleted: e6391432e12cf7305a906d4264230c98cfe04f9a96ba2d97305964ad1b709986
Deleted: a3c0f7711c5e6b6f9181958c18daa56809bfa45a03f3a45da4cd0d83675dd896
Deleted: c8de9902faf0bd04893447386c8aeeb1e551b3b6622470f4eb838428359672d7
Deleted: cf0a75889057c9255505ce5ef36a0d00ac480c945557adfb12674417a0d3efed
Untagged: <privateDockerHub>/linux:ubuntu-14.04
Untagged: <privateDockerHub>/linux@sha256:0c2a0ba12cb5bd285cb12abc4a6ce905a8271cc1f94fd7b24d33ad229f7a1a9a
Deleted: a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 => deleted multiple times
Deleted: 7fe6916ab38229dcf39f8534843008d42dd93a44241619505dbf4774f0b70d28
Deleted: 86656bbaa6fd4511807a37afa01c1d4dce5cc973faed463aafa4fabd32b6dcde
Deleted: dc9874b529521c67727d1a5ad5d8eb24af93cdc5aa232cc42ff37488c4c2e5c8
Deleted: 65584f5f70eeea5b72e357d4e2bc0edf9b1a82fb23f1de65880e1dae719f78ab
Deleted: 72c01b436656c9a55ae968ed14e4f1b2a36e11a1103de1d78052edc926d5003f
Untagged: ubuntu:14.04
Untagged: ubuntu@sha256:5545d04a2ad5de4b5a13a97ce603539b24679292a897f5db36038879b139c05f

This was a case that still worked, but if I try to download a image with maybe 10 layers more, this will crash with following error.

Logs

Failed to write to image store: [POST /storage/{store_name}][500] WriteImage default &{Code:500 Message:ServerFaultCode: XML element depth exceeds configured maximum 100

Stacktrace while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 45390 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 44909 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 44428 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 43947 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 43466 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 42985 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 42504 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 42023 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 41542 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 41061 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 40580 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 40099 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 39618 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 39137 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 38656 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 38175 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 37694 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 37213 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 36732 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 36251 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 35770 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 35289 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 34808 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 34327 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 33846 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 33365 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 32884 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 32403 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 31922 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 31441 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 30960 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 30479 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 29998 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 29517 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 29036 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 28555 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 28074 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 27593 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 27112 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 26631 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 26150 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 25669 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 25188 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 24707 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 24226 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 23745 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 23264 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 22783 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 22302 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 21821 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 21340 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 20859 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 20378 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 19897 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 19416 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 18935 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 18454 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 17973 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 17492 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 17011 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 16530 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 16049 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 15568 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 15087 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 14606 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 14125 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 13644 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 13163 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 12682 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 12201 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 11720 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 11239 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 10758 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 10277 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 9796 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 9315 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 8834 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 8353 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 7872 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 7391 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 6910 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 6429 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 5948 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 5467 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 4986 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 4505 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 4024 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 3543 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 3062 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 2581 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 2100 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 1619 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 1138 while parsing property "parent" of static type VirtualDiskFlatVer2BackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo at line 2, column 545 while parsing property "backing" of static type VirtualDeviceBackingInfo while parsing serialized DataObject of type vim.vm.device.VirtualDisk at line 2, column 364 while parsing property "device" of static type VirtualDevice while parsing serialized DataObject of type vim.vm.device.VirtualDeviceSpec at line 2, column 202 while parsing property "deviceChange" of static type ArrayOfVirtualDeviceConfigSpec while parsing serialized DataObject of type vim.vm.ConfigSpec at line 2, column 196 while parsing call information for method ReconfigVM_Task at line 2, column 118 while parsing SOAP body at line 2, column 112 while parsing SOAP envelope at line 2, column 0 while parsing HTTP request for method reconfigure on object of type vim.VirtualMachine at line 1, column 0}

See also

https://github.com/vmware/vic/issues/8153 https://github.com/vmware/vic/issues/524#issuecomment-237719105

renmaosheng commented 6 years ago

Hi Kim, thanks for reporting the issue, based on our observation, VIC can support image less than 90 layers. (https://github.com/vmware/vic-product/pull/1980), could you please share the image that causes the issue with only 10 layers more? thanks.

kimbamps commented 6 years ago

Hello

I was able to create a simple container based on Ubuntu 14.04 with only 25 layers to reproduce the problem. kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-101

From what I understand of it now. It looks to me that VIC only support images with less then 100 layers like they where defined in the past (pre Docker v1.10). This article (https://windsock.io/explaining-docker-image-ids/) gave me some nice clues and insides about layers.

I created 3 images starting from Ubuntu image with only a lot of fake labels and arguments and 20 RUN's with a mkdir. If you run the following commands you can see that they all have 25 layers.

$ docker image inspect kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-88 -f '{{.RootFS.Layers}}' | wc -w
25
$ docker image inspect kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-93 -f '{{.RootFS.Layers}}' | wc -w
25
$ docker image inspect kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-101 -f '{{.RootFS.Layers}}' | wc -w
25

If you use docker histroy to see how the layers where created in the past. You can see that there are 2 images below 100 and one above. The first 2 are working, the last is not and giving the same error as in my first post.

$ docker history kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-88 | wc -l
88
$ docker history kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-93 | wc -l
93
$ docker history kimbamps/vic-layer-issue:ubuntu-14.04-layers-25-101 | wc -l
101

When pulling these images you can see the same behavior as I described i my first post. I appears we get and "empty" layer for each LABEL, ARG, ... This also explains why we get the message that we exceed the maximum of 100...

renmaosheng commented 6 years ago

Hi Kim， The limitation is just like what you described. We have some vsphere API limitation of generating larger than 100 layers, best practice is to reduce the unnecessary layers during the image generation process while you write the Dockerfile. Thanks, Steven

kimbamps commented 6 years ago

Steven

Don't we see this behaviour in VIC because it is still using the old way of looking at image layers? Can't this be improved by upgrading the docker daemon in the VCH? So that vic is also looking at layers like the newer docker daemons do? Or am I understanding some things wrongly here?

If this is the case, do you know if there are concreet plans to improve this issue/limitation? Because if every "command" in the Dockerfile's counts, you will easily reach 100. Especially in more complex environments. We have multiple base layer from where a whole range of Docker images are created. So that we can reuse as much as possible. And every layer had at least following commands. FROM, LABEL, MAINTAINER So they will all be seen as extra "empty" layers for each base image...

The general structure of base layers look like this; Ubuntu -> updated Ubuntu with date, language settings, ... -> A layer with several shared bash scripts -> JDK layer (several versions) -> tomcat layer (several version) -> ...

Currently this limitation is a blokker for us. :(

zjs commented 6 years ago

Don't we see this behaviour in VIC because it is still using the old way of looking at image layers?

Yes, essentially. I believe this could be fixed following the approach described in #5960.

If this is the case, do you know if there are concreet plans to improve this issue/limitation?

This work has not yet been scheduled for a specific release, but I believe you're the first user to bring up this issue — previous suggestions for improvements here have just come from within the development team. The information you've provided will certainly help us when determining the priority of this work relative to other projects. Thanks for the filing this issue!

cc @clouderati, the product manager for VIC