search

TritonDataCenter / sdc-docker

Docker Engine for Triton
Mozilla Public License 2.0
182 stars 49 forks source link

zfs_snapshot_tar error when building a docker container on a centos7 image #100

Open pccowboy opened 7 years ago

pccowboy commented 7 years ago

Running a centos 7 image as a docker host on latest Triton. Trying to build a docker image with a specific Perl to match.

When I build using the below dockerfile, I get an error as it appears to be working on the final layer:

Build failed: zfs_snapshot_tar exited with code: 1 (zfs_snapshot_tar: zfs exited non-zero (1) with: Unable to determine path or stats for object 57391 in zones/1b4ddc80-02d0-47bc-a26e-44b00a60da7f@buildlayer7: Invalid argument) (req_id: dfc56840-acea-11e6-9ed1-3d894f48c694)

Then, the next attempt I make to call build gets a second error, telling me some unknown image is not available.

build.log.txt

The full 'docker build' log is attached.

After I break the docker file into multiple, smaller bits, and build them in order, I can get a successful build. I have attached the bits as well.

Dockerfile that fails:

FROM centos:centos7

RUN yum -y update && yum -y groupinstall 'Development Tools'
RUN yum -y install \
    kernel-devel \
    ImageMagick-devel \
    bzip2-devel \
    bzr \
    mecurial \
    expat-devel \
    libcurl \
    libcurl-devel \
    openssl-devel \
    libevent-devel \
    libffi-devel \
    glib2-devel \
    libjpeg-devel \
    mysql-devel \
    postgresql-devel \
    ncurses-devel \
    readline \
    readline-devel \
    sqlite-devel \
    openssl \
    openssl-devel \
    libxml2-devel \
    libxslt-devel \
    zlib-devel \
    libyaml-devel \
    wget \
    tar \
    && yum clean all

RUN mkdir /usr/src/perl
COPY *.patch /usr/src/perl/
WORKDIR /usr/src/perl

RUN curl -SL https://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.16.3.tar.bz2 -o perl-5.16.3.tar.bz2 \
    && echo '060bc17cf9f142d043f9bf7b861422ec624875ea *perl-5.16.3.tar.bz2' | sha1sum -c - \
    && tar --strip-components=1 -xjf perl-5.16.3.tar.bz2 -C /usr/src/perl \
    && rm perl-5.16.3.tar.bz2 \
    && cat *.patch | patch -p1 \
    && ./Configure -Duse64bitall -Duseshrplib -A ccflags=-fwrapv -des \
    && make -j$(nproc) \
    && make test_harness \
    && make install \
    && cd /usr/src \
    && curl -LO https://raw.githubusercontent.com/miyagawa/cpanminus/master/cpanm \
    && chmod +x cpanm \
    && ./cpanm App::cpanminus \
    && rm -fr ./cpanm /root/.cpanm /usr/src/perl/* /tmp/*

Dockerfiles that work: first - " docker --tls build -t zettit/centos ."

FROM centos:centos7

RUN yum -y update && yum -y groupinstall 'Development Tools'
RUN yum -y install \
    kernel-devel \
    ImageMagick-devel \
    bzip2-devel \
    bzr \
    mecurial \
    expat-devel \
    libcurl \
    libcurl-devel \
    openssl-devel \
    libevent-devel \
    libffi-devel \
    glib2-devel \
    libjpeg-devel \
    mysql-devel \
    postgresql-devel \
    ncurses-devel \
    readline \
    readline-devel \
    sqlite-devel \
    openssl \
    openssl-devel \
    libxml2-devel \
    libxslt-devel \
    zlib-devel \
    libyaml-devel \
    wget \
    tar

Second - "docker --tls build -t zettit/centos_perl .":

FROM zettit/centos

RUN mkdir /usr/src/perl
COPY *.patch /usr/src/perl/
WORKDIR /usr/src/perl

RUN curl -SL https://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.16.3.tar.bz2 -o perl-5.16.3.tar.bz2 \
    && echo '060bc17cf9f142d043f9bf7b861422ec624875ea *perl-5.16.3.tar.bz2' | sha1sum -c - \
    && tar --strip-components=1 -xjf perl-5.16.3.tar.bz2 -C /usr/src/perl

RUN rm perl-5.16.3.tar.bz2

RUN cat *.patch | patch -p1 \
    && ./Configure -Duse64bitall -Duseshrplib -A ccflags=-fwrapv -des \
    && make -j$(nproc) \
    && make test_harness

RUN make install

RUN mkdir /usr/src/cpanm
WORKDIR /usr/src/cpanm
RUN cd /usr/src/cpanm \
    && curl -LO https://raw.githubusercontent.com/miyagawa/cpanminus/master/cpanm \
    && chmod +x cpanm \
    && ./cpanm App::cpanminus

RUN rm -fr ./cpanm /root/.cpanm /usr/src/perl /tmp/*
twhiteman commented 7 years ago

Thanks for reporting this. I don't immediately see what is causing this problem, and I've not been able to reproduce this issue yet - here's my successful build of this Dockerfile (though I faked the patches by just adding one empty.patch file - so the COPY succeeds): https://gist.github.com/twhiteman/50666658044b6ca52f8b2ef427849cfe

Could you retry by adding the --no-cache option to docker build, e.g.: docker --tls build --no-cache .

pccowboy commented 7 years ago

I did try that option previously, no difference with the single file.

I have been adding packages to the smaller files as I develop, and I am running into the same error again, even with the --no-cache=true option.

How would I determine what object it cannot find - is that a dtrace-level debugging, or is there info already in place I can dig out?

pccowboy commented 7 years ago

FWIW, this is what appears when I try a build command immediately after the failure - I see this on the build host:

Build failed: zfs_snapshot_tar exited with code: 1 (zfs_snapshot_tar: zfs exited non-zero (1) with: Unable to determine path or stats for object 57419 in zones/6bcc1f40-3e4f-4d73-b2df-cca2209d12d9@buildlayer9: Invalid argument

) (req_id: 5621f570-b3fe-11e6-a236-e76d7db7c3b9)
[dswift@0ff275d0-37c9-4cb3-8c27-c9373f3d4b50 rss_reader]$ docker --tls build  -t zettit/centos_perl centos_perl  2>&1 | tee -a build.log                                                                                                            Sending build context to Docker daemon 15.87 kB
Build failed: No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4 (req_id: 523b6fa0-b402-11e6-a236-e76d7db7c3b9)

and this in the docker servcie log

{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c
3b9","level":30,"evt":{"ph":"b","name":"build","req_seq":1480183274393003},"msg":"","time":"2016-11-26T18:01:14.394Z","v":
0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"level":30,"msg":"UFDS reconnected","time":
"2016-11-26T18:01:14.530Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"component":"ufds","clazz":"Client","ldap_i
d":"3__ldaps://ufds.home.pccowboy.com","level":20,"msg":"connected after 1 attempt(s)","time":"2016-11-26T18:01:14.530Z","
v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c
3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"msg":"req.query:  { buildargs: '{}'
,\n  cgroupparent: '',\n  cpuperiod: '0',\n  cpuquota: '0',\n  cpusetcpus: '',\n  cpusetmems: '',\n  cpushares: '0',\n  do
ckerfile: 'Dockerfile',\n  labels: '{}',\n  memory: '0',\n  memswap: '0',\n  rm: '1',\n  shmsize: '0',\n  t: 'zettit/cento
s_perl',\n  ulimits: 'null' }","time":"2016-11-26T18:01:14.578Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c
3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"msg":"rat:  { index: { name: 'docke
r.io', official: true },\n  official: false,\n  remoteName: 'zettit/centos_perl',\n  localName: 'zettit/centos_perl',\n  c
anonicalName: 'docker.io/zettit/centos_perl',\n  tag: 'latest' }","time":"2016-11-26T18:01:14.578Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c
3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"msg":"getScratchImage: found scratc
h image in IMGAPI","time":"2016-11-26T18:01:14.585Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c
3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"msg":"scratchImage:  { architecture
: 'amd64',\n  author: '',\n  comment: '',\n  created: 1448971200000,\n  docker_id: '0000000000007363726174636800aeee14d7ea
bd60b564604d05000000000000',\n  head: true,\n  heads: [],\n  image_uuid: '4728c3f7-5fc9-14b3-0e9b-3b241ec89745',\n  index_
name: 'docker.io',\n  owner_uuid: undefined,\n  private: true,\n  size: 0,\n  virtual_size: 0,\n  container_config: \n   {
 Hostname: '',\n     Domainname: '',\n     User: '',\n     AttachStdin: false,\n     AttachStdout: false,\n     AttachStde
rr: false,\n     ExposedPorts: null,\n     PublishService: '',\n     Tty: false,\n     OpenStdin: false,\n     StdinOnce: false,\n     Env: null,\n     Cmd: null,\n     Image: '',\n     Volumes: null,\n     VolumeDriver: '',\n     WorkingDir: '',\n     Entrypoint: null,\n     NetworkDisabled: false,\n     MacAddress: '',\n     OnBuild: null },\n  config: \n   { Hostname: '',\n     Domainname: '',\n     User: '',\n     AttachStdin: false,\n     AttachStdout: false,\n     AttachStderr: false,\n     ExposedPorts: null,\n     PublishService: '',\n     Tty: false,\n     OpenStdin: false,\n     StdinOnce: false,\n     Env: null,\n     Cmd: null,\n     Image: '',\n     Volumes: null,\n     VolumeDriver: '',\n     WorkingDir: '',\n     Entrypoint: null,\n     NetworkDisabled: false,\n     MacAddress: '',\n     OnBuild: null } }","time":"2016-11-26T18:01:14.586Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"imgName":"068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4","img":{"author":"","architecture":"amd64","comment":"","created":1480183042184,"config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/bash"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"container_config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","make install"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"docker_id":"068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4","head":false,"image_uuid":"f53bc24c-827a-a8d2-3715-6b8067e119bc","index_name":"docker.io","owner_uuid":"1eee2213-b3c0-ea88-8609-f2bcace05318","parent":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","private":true,"heads":["3fc4a2dbfeff6f5bbf0c945aebffb85ebc5261ff8bc1ce5eac08e376e99a550d"],"size":589248,"virtual_size":1205265864},"msg":"imgFromName: findImage","time":"2016-11-26T18:01:14.851Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"imgIsGone":true,"img":{"author":"","architecture":"amd64","comment":"","created":1480183042184,"config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/bash"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"container_config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","make install"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"docker_id":"068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4","head":false,"image_uuid":"f53bc24c-827a-a8d2-3715-6b8067e119bc","index_name":"docker.io","owner_uuid":"1eee2213-b3c0-ea88-8609-f2bcace05318","parent":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","private":true,"heads":["3fc4a2dbfeff6f5bbf0c945aebffb85ebc5261ff8bc1ce5eac08e376e99a550d"],"size":589248,"virtual_size":1205265864},"msg":"imgFromName: delImgRef","time":"2016-11-26T18:01:14.896Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"params":{"author":"","architecture":"amd64","comment":"","created":1480183042184,"config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/bash"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"container_config":{"Hostname":"d6dcf178f680","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","make install"],"Image":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","Volumes":null,"WorkingDir":"/usr/src/perl","Entrypoint":null,"OnBuild":null,"Labels":{"build-date":"20161102","license":"GPLv2","name":"CentOS Base Image","vendor":"CentOS"}},"docker_id":"068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4","head":false,"image_uuid":"f53bc24c-827a-a8d2-3715-6b8067e119bc","index_name":"docker.io","owner_uuid":"1eee2213-b3c0-ea88-8609-f2bcace05318","parent":"b3202f510dcbccbbb447e31ec224cbc96a3a0403c4fd42ba81eae298314307ee","private":true,"heads":["3fc4a2dbfeff6f5bbf0c945aebffb85ebc5261ff8bc1ce5eac08e376e99a550d"],"size":589248,"virtual_size":1205265864},"msg":"deleteImage: entry","time":"2016-11-26T18:01:14.896Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":20,"msg":"build: final callback, err: {\"message\":\"No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4\",\"statusCode\":404,\"body\":{\"code\":\"ResourceNotFound\",\"message\":\"No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4\"},\"restCode\":\"ResourceNotFound\"}","time":"2016-11-26T18:01:14.910Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","level":50,"msg":"docker build error { [ResourceNotFoundError: No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4]\n  message: 'No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4',\n  statusCode: 404,\n  body: \n   { code: 'ResourceNotFound',\n     message: 'No such image: 068f64b587086f038e75e18d4da029a4a3a6c801669061d9ca01876c95a338a4' },\n  restCode: 'ResourceNotFound' }","time":"2016-11-26T18:01:14.910Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","level":30,"evt":{"ph":"e","name":"build","req_seq":1480183274393003},"msg":"","time":"2016-11-26T18:01:14.910Z","v":0}
{"name":"docker","hostname":"8b39a892-64a0-4b39-afaa-584932ff8ad4","pid":10569,"req_id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","account":"1eee2213-b3c0-ea88-8609-f2bcace05318","login":"zettit_ops","component":"audit","route":"build","audit":true,"level":30,"remoteAddress":"10.67.42.27","remotePort":48369,"req":{"method":"POST","url":"/v1.24/build?buildargs=%7B%7D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&rm=1&shmsize=0&t=zettit%2Fcentos_perl&ulimits=null","headers":{"host":"10.67.42.6:2376","user-agent":"Docker-Client/1.12.3 (linux)","transfer-encoding":"chunked","content-type":"application/tar","x-registry-config":"(censored)","accept-encoding":"gzip"},"httpVersion":"1.1","trailers":{},"version":"*","timers":{"handler-0":60,"bunyan":99,"handler-2":393,"checkReadonlyMode":21,"checkServices":41,"reqAuth":182997,"reqClientApiVersion":35,"parseQueryString":310,"build":332892}},"res":{"statusCode":200,"headers":{"date":"Sat, 26 Nov 2016 18:01:14 GMT","x-request-id":"523b6fa0-b402-11e6-a236-e76d7db7c3b9","x-response-time":185,"server":"Triton/1.9.0 (linux)"},"trailer":false},"err":true,"latency":518,"_audit":true,"msg":"handled: 200","time":"2016-11-26T18:01:14.911Z","v":0}

The 'No such image' failures continue to appear until I use the '--no-cache==true' flag

pccowboy commented 7 years ago

I had thought it might be related, but then I removed those lines, to try and narrow down what might be causing the problem. It still occurs without those lines at the end of the Dockerfile.

Chayeon Hong mailto:notifications@github.com Sunday, November 27, 2016 5:37 PM

Do you think this issue related to

The reason for my thought like this is the last line in the Dockerfile.

RUN rm -fr ./cpanm /root/.cpanm /usr/src/perl /tmp/*

Thank you.

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joyent/sdc-docker/issues/100#issuecomment-263164717, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkG3etzq3tVtmadTb-wnBrUDtXppqE7ks5rCjBXgaJpZM4K5Ka4.

twhiteman commented 7 years ago

@pccowboy sounds like a zfs diff error (which is what the zfs_snapshot_tar is based on top of).

To narrow this down, you'll probably want to do some tweaking to the cn-agent (which is on the compute node - or headnode if using COAL).

  1. Modify /opt/smartdc/agents/lib/node_modules/cn-agent/bin/docker-build.js and comment out the call to destroySnapshots. This will leave behind the zfs snapshots that are created during the build (no need to restart any services).

  2. Run docker build again - also add --rm=0 to your docker build command line (this may not be needed, as on error I think the container is already left there).

  3. Look at the last docker build child log - in /var/log/cn-agent/logs directory (on the compute node), which will show the zfs_snapshot_tar command line, to see which snapshots are failing.

  4. Run zfs diff zones/${UUID}@buildlayerX zones/${UUID}@buildlayerY - does it fail?