search

slimtoolkit / slim

Slim(toolkit): Don't change anything in your container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)
Apache License 2.0
19.48k stars 730 forks source link

FATA[0000] docker-slim: failure #23

Open kklepper opened 8 years ago

kklepper commented 8 years ago

What am I to do with this? profile result is the same as build or info result.

$ docker -v
Docker version 1.11.1, build 5604cbe
$ docker-slim -v
docker-slim version Tetra|1.14-5-gd4ced16|d4ced16610f713644f996a8eb8a61d7cf4b1c4e7|2016-03-14_01:16:22AM

Sample 1:

$ docker-slim profile kklepper/yaws:ubuntu
docker-slim: [profile] image=kklepper/yaws:ubuntu
INFO[0000] docker-slim: inspecting 'fat' image metadata...
FATA[0000] docker-slim: failure                          error=stat /usr/local/bin/.images/41f230937cf1da22a7b7bda7e1c06ae4841e39ee0c261cd0a8f228524b53a1b9/artifacts: no such file or directory stack=/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/errors.go:11 (0x558ad6)
/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/dirs.go:368 (0x558837)
/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/master/commands/profile.go:43 (0x4b0191)
/docker-slim/apps/docker-slim/cli.go:395 (0x40c05d)
/docker-slim/_vendor/src/github.com/codegangsta/cli/command.go:174 (0x4bfb47)
/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:187 (0x4bc0b5)
/docker-slim/apps/docker-slim/cli.go:507 (0x405f6b)
/docker-slim/apps/docker-slim/main.go:5 (0x406059)
/usr/local/go/src/runtime/proc.go:111 (0x4389c0)
        main:
/usr/local/go/src/runtime/asm_amd64.s:1721 (0x468981)
        goexit: JB loop5to7

Sample 2:

$ docker-slim profile kklepper/maxscale-1.4.3:centos
docker-slim: [profile] image=kklepper/maxscale-1.4.3:centos
INFO[0000] docker-slim: inspecting 'fat' image metadata...
FATA[0000] docker-slim: failure                          error=stat /usr/local/bin/.images/dcca93f57ff1d50051ed2f1170c5f03ab6abf838e477a7b4c28693cffdcc4426/artifacts: no such file or directory stack=/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/errors.go:11 (0x558ad6)
/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/dirs.go:368 (0x558837)
/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/master/commands/profile.go:43 (0x4b0191)
/docker-slim/apps/docker-slim/cli.go:395 (0x40c05d)
/docker-slim/_vendor/src/github.com/codegangsta/cli/command.go:174 (0x4bfb47)
/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:187 (0x4bc0b5)
/docker-slim/apps/docker-slim/cli.go:507 (0x405f6b)
/docker-slim/apps/docker-slim/main.go:5 (0x406059)
/usr/local/go/src/runtime/proc.go:111 (0x4389c0)
        main:
/usr/local/go/src/runtime/asm_amd64.s:1721 (0x468981)
        goexit: JB loop5to7
kcq commented 8 years ago

Looks like the docker-slim binaries are in /usr/local/bin and it's not writable, so you get this failure. The master app needs a place to keep its state (in the ".images" directory). By default it happens to be the same directory where the executable is located.

You have two options:

  1. Put the docker-slim binaries somewhere else where the directory is writable.
  2. Wait for the upcoming version (hopefully later today) that will allow you to specify an alternative state location.

Sorry for the mix up. Need to do a better job with the documentation :-)

kcq commented 8 years ago

The latest release (1.15: darwin|Tetra|1.15|98b6913d1811004548e7989310c8b8c02da6bdbb|2016-06-20_05:02:53AM) has the --state-path parameter to set an alternative state location.

By the way, what's in the first container (yaws)? If it's the Erlang-based web server I'm very curious how it's configured and what's included. DockerSlim hasn't been tested with Erlang services yet, so there might be other unexpected results. Let me know what happens and what other errors you get there.

Thanks again for the report!

kklepper commented 8 years ago

Thanks for your excellent work! I tested with my Yaws container:

ubuntu@ip-172-31-30-16:~/kklepper/yaws$ docker images | grep yaws
kklepper/yaws.slim                   latest              60b5320fbfff        9 minutes ago       9.747 MB
kklepper/yaws                        ubuntu              41f230937cf1        3 weeks ago         942.5 MB

This result looks pretty much like too good to be true. On the other hand, I have a couple of Alpine containers which do serious work (haproxy, ssh, memcached) and aren't bigger than that, so maybe I'm lucky. The slim container starts fine, but doesn't work, unfortunately, as it should. 

ubuntu@ip-172-31-30-16:~/kklepper/yaws$ curl localhost
curl: (56) Recv failure: Connection reset by peer

The log file doesn't show any errors, though. As ps is not implemented, I am out of ideas of how to test Yaws from within the container.

This is the Yaws Dockerfile which I was unable to transfer to Alpine Linux.

FROM ubuntu:14.04

MAINTAINER kklepper 

RUN echo 'APT::Install-Recommends 0;' >> /etc/apt/apt.conf.d/01norecommends \
 && echo 'APT::Install-Suggests 0;' >> /etc/apt/apt.conf.d/01norecommends \
 && apt-get update \
 && apt-get build-dep -y erlang yaws \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    vim.tiny \
    wget \
    sudo \
    net-tools \
    ca-certificates \
    unzip \
    erlang-nox \
    erlang-src \
    erlang-manpages \
    erlang-mode \
    erlang-dev \
    libtool \
    automake \
    git-core \
 && rm -rf /var/lib/apt/lists/* \
 && git clone git://github.com/klacke/yaws.git

WORKDIR yaws

RUN autoreconf -fi \
 && ./configure --help \
 && ./configure --localstatedir=/var --sysconfdir=/etc \
 && make \
 && make install

RUN apt-get update \
 && apt-get install -y php5-cgi

ENV VER 0.2

VOLUME /wp/ci

COPY ./files/favicon.ico /yaws/www/favicon.ico
COPY ./files/start.sh /yaws/
COPY ./etc/yaws.conf /etc/yaws/yaws.conf
COPY ./voxx /usr/local/lib/yaws/voxx

CMD ["/yaws/start.sh"]

In my scenario, Yaws acts as a proxy to NGINX utilizing the Mnesia database for caching. To this end, I have written a couple of Erlang files residing in yaws/voxx.

This is what I get from my 2nd sample, another fat container which I would like to slim-down:

# ubuntu@ip-172-31-30-16:/tmp/dist_linux$ ./docker-slim build --http-probe kklepper/maxscale-1.4.3:centos
docker-slim: [build] image=kklepper/maxscale-1.4.3:centos http-probe=true remove-file-artifacts=false image-overrides=map[] entrypoint=[] (false) cmd=[] (false) workdir='' env=[] expose=map[]
INFO[0000] docker-slim: inspecting 'fat' image metadata...
INFO[0000] docker-slim: [sha256:dcca93f57ff1d50051ed2f1170c5f03ab6abf838e477a7b4c28693cffdcc4426] 'fat' image size => 303829120 (304 MB)

INFO[0000] docker-slim: processing 'fat' image info...
INFO[0000] docker-slim: starting instrumented 'fat' container...
INFO[0000] docker-slim: created container => 4ed1f015d7bf16526aa32c67ee82c0657e89efc23cebbf71aedb4f44137d2b3f
INFO[0000] docker-slim: watching container monitor...
docker-slim: press <enter> when you are done using the container...
INFO[0004] docker-slim: HTTP probe started...
INFO[0004] docker-slim: http probe - GET http://127.0.0.1:33323/ error: Get http://127.0.0.1:33323/: EOF

INFO[0009] docker-slim: http probe - GET http://127.0.0.1:33322/ error: Get http://127.0.0.1:33322/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

INFO[0009] docker-slim: http probe - GET http://127.0.0.1:33325/ error: Get http://127.0.0.1:33325/: EOF

INFO[0009] docker-slim: http probe - GET http://127.0.0.1:33324/ error: Get http://127.0.0.1:33324/: EOF

INFO[0009] docker-slim: HTTP probe done.

INFO[0037] docker-slim: waiting for the container finish its work...

INFO[0041] docker-slim: shutting down 'fat' container...
INFO[0042] docker-slim: processing instrumented 'fat' container info...
INFO[0042] docker-slim: generating AppArmor profile...
INFO[0042] docker-slim: building 'slim' image...
FATA[0042] docker-slim: failure                          error=no permission to read from '/tmp/dist_linux/.images/dcca93f57ff1d50051ed2f1170c5f03ab6abf838e477a7b4c28693cffdcc4426/artifacts/files/run/maxscale/maxscale.pid' stack=/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/errors.go:11 (0x561576)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/master/commands/build.go:135 (0x4aeccb)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/cli.go:321 (0x40b623)
/usr/local/go/src/runtime/asm_amd64.s:437 (0x466dce)
        call32: JMP     AX
/usr/local/go/src/reflect/value.go:432 (0x53fc4a)
        Value.call: }
/usr/local/go/src/reflect/value.go:300 (0x53e911)
        Value.Call: func (v Value) Call(in []Value) []Value {
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:483 (0x4bfcce)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/command.go:186 (0x4c1c7f)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:237 (0x4bd369)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/cli.go:526 (0x4063fb)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/main.go:5 (0x4064e9)
/usr/local/go/src/runtime/proc.go:111 (0x439140)
        main:
/usr/local/go/src/runtime/asm_amd64.s:1721 (0x469101)
        goexit: JB loop5to7

Sample 3:

ubuntu@ip-172-31-30-16:/tmp/dist_linux$ ./docker-slim build --http-probe kklepper/mariadb-10.1-ms-mha:alpine
docker-slim: [build] image=kklepper/mariadb-10.1-ms-mha:alpine http-probe=true remove-file-artifacts=false image-overrides=map[] entrypoint=[] (false) cmd=[] (false) workdir='' env=[] expose=map[]
INFO[0000] docker-slim: inspecting 'fat' image metadata...
INFO[0000] docker-slim: [sha256:b22539d0816942fe1db90e75363915dec18fcb9f87f8d87f83072eff3cdc4fa0] 'fat' image size => 272783598 (273 MB)

INFO[0000] docker-slim: processing 'fat' image info...
INFO[0000] docker-slim: starting instrumented 'fat' container...
INFO[0000] docker-slim: created container => bb23baf04bcd0a000a15116b16e7354994c6eadc7cd7333cfb3547e55d38475e
INFO[0000] docker-slim: watching container monitor...
docker-slim: press <enter> when you are done using the container...

INFO[0003] docker-slim: waiting for the container finish its work...
INFO[0003] docker-slim: shutting down 'fat' container...
INFO[0003] docker-slim: processing instrumented 'fat' container info...
INFO[0003] docker-slim: generating AppArmor profile...
INFO[0003] docker-slim: building 'slim' image...
Step 1 : FROM scratch
 --->
Step 2 : COPY files /
FATA[0003] docker-slim: failure                          error=lstat files: no such file or directory stack=/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/utils/errors.go:11 (0x561576)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/cloudimmunity/docker-slim/master/commands/build.go:135 (0x4aeccb)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/cli.go:321 (0x40b623)
/usr/local/go/src/runtime/asm_amd64.s:437 (0x466dce)
        call32: JMP     AX
/usr/local/go/src/reflect/value.go:432 (0x53fc4a)
        Value.call: }
/usr/local/go/src/reflect/value.go:300 (0x53e911)
        Value.Call: func (v Value) Call(in []Value) []Value {
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:483 (0x4bfcce)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/command.go:186 (0x4c1c7f)
/Users/me/Desktop/CI_GITHUB/docker-slim/_vendor/src/github.com/codegangsta/cli/app.go:237 (0x4bd369)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/cli.go:526 (0x4063fb)
/Users/me/Desktop/CI_GITHUB/docker-slim/apps/docker-slim/main.go:5 (0x4064e9)
/usr/local/go/src/runtime/proc.go:111 (0x439140)
        main:
/usr/local/go/src/runtime/asm_amd64.s:1721 (0x469101)
        goexit: JB loop5to7

I hope this helps.

kklepper commented 8 years ago

Here is a crash dump of yaws:

ubuntu@ip-172-31-30-16:/tmp/dist_linux$ docker run -it --rm -v /tmp:/tmp --name u2 kklepper/yaws.slim sh
# yaws -i
{error_logger,{{2016,6,20},{15,49,9}},crash_report,[[{initial_call,{supervisor_bridge,user_sup,['Argument__1']}},{pid,<0.23.0>},{registered_name,[]},{error_info,{exit,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,99}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,48}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,79}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[kernel_sup,<0.10.0>]},{messages,[]},{links,[<0.11.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,150}],[]]}
{error_logger,{{2016,6,20},{15,49,9}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,99}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,48}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,79}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{offender,[{pid,undefined},{name,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]}]}
{error_logger,{{2016,6,20},{15,49,9}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,user,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,99}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,48}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,79}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,147}],[]]}
{error_logger,{{2016,6,20},{15,49,9}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,user,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,99}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,48}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,79}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,user,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,\"user_sup.erl\"},{line,99}]},{user_sup,init,1,[{file,\"user_sup.erl\"},{line,48}]},{supervisor_bridge,init,1,[{file,\"supervisor_bridge.erl\"},{line,79}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}}},{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,user,{undef,[{user_drv,start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},

Does this tell you something?

q-cloudlock commented 8 years ago

It'll take a bit to get to the bottom of this :-)

I'm curious... what's in /yaws/start.sh? What port is Yaws running on? What endpoints are available? It'll be good to configure the http probe to ping some of those endpoints (even if they are proxy endpoints).

You can also use the '--include-path' parameter to include extra files and directories in the minified image, so it's easier to debug it.

Thank you for spending extra time trying to troubleshoot the problems. I greatly appreciate it!

kklepper commented 8 years ago

I highly appreciate your efforts and gladly contribute to the success of your project if I can.

/yaws/start.sh

#!/bin/sh
cd /yaws 
yaws --daemon
tail -f /var/log/apt/history.log

So this is straightforward, tail just keeps the container running.

The crucial section from yaws.conf:

<server localhost>
        port = 80
        listen = 0.0.0.0
        docroot = "/ci"
        auth_log = true
        errormod_404 = out404
        php_handler = <cgi, "/usr/bin/php5-cgi">
        deflate = true
        appmods = </, myurl exclude_paths plugins img images styles scripts>
</server>

myurl is an appmod I have written which checks for an entry in the mnesia database first, else redirects to nginx running on port 8010 and caches the result in mnesia. Seeing this, I wonder why I need that php_handler directive... same with docroot - yaws does nothing with it, nginx needs that, of course. I remember, though, that I ran into problems when docroot was set wrongly. Maybe yaws just needs something valid here in order to run. out404 just spits out a meaningful error message.

Maybe we should ask for hints from yaws experts via their list to understand what happened - erlang error messages are meaningful but not easy to understand ... what do you think?

I'm afraid I am not able to test as you would like. What about the other cases?