Open ProfessorManhattan opened 3 years ago
kcq
commented
3 years ago Thanks for the repro details @ProfessorManhattan ! It's super useful. Looks like the main process can't connect to the temporary container it created. It's possible that the target container crashed. Do you mind adding the --show-clogs build command flag to see the logs from the temporary container that gets created. Also can you try using the 1.35.2 version of docker-slim.
ProfessorManhattan
commented
3 years ago Hey @kcq -- thanks for following up. I added --show-clogs and upgraded to 1.35.2 but that did not work.
Things I have tried:
Here's the build with TLS enabled:
https://gitlab.com/megabyte-labs/dockerfile/ci-pipeline/ansible-lint/-/jobs/1264169021
And here's the build with TLS disabled (basically the same thing):
https://gitlab.com/megabyte-labs/dockerfile/ci-pipeline/dockerslim/-/jobs/1284746324
Output using --show-clogs=true instead of just --show-clogs:
https://gitlab.com/megabyte-labs/dockerfile/ci-pipeline/dockerslim/-/jobs/1284791596
And here's the updated container I'm using:
FROM docker:20.10.6 as build
ENV container docker
ENV DOCKERSLIM_SHA256 d87d6c52ccc39f2823817cdd29951eb38237fe2220572558a9281bb37800155c
ENV DOCKERSLIM_URL https://downloads.dockerslim.com/releases/1.35.2/dist_linux.tar.gz
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
RUN apk --no-cache add --virtual build-dependencies \
upx=3.96-r1 \
wget=1.21.1-r1 \
&& wget -nv $DOCKERSLIM_URL -O /tmp/dockerslim.tar.gz \
&& echo "$DOCKERSLIM_SHA256 /tmp/dockerslim.tar.gz" | sha256sum -c \
&& tar -xzvf /tmp/dockerslim.tar.gz \
&& cp -rf dist_linux/* /usr/local/bin \
&& rm -rf /tmp/* dist_linux \
&& upx /usr/local/bin/docker-slim \
&& upx /usr/local/bin/docker-slim-sensor
FROM docker:20.10.6
COPY --from=build /usr/local/bin/docker-slim /usr/local/bin/docker-slim
COPY --from=build /usr/local/bin/docker-slim-sensor /usr/local/bin/docker-slim-sensor
RUN apk --no-cache add \
jq=1.6-r1 \
&& rm -Rf /var/cache/apk/*
WORKDIR /workThanks for looking at this. This is my favorite project on GitHub and I've looked through the top 10,000 projects sorted by stars.
This might be a GitLab CI specific issue. They require you to run two images at the same time and specify one as a service. I'm not sure why but maybe that has something to do with this issue.
ProfessorManhattan
commented
3 years ago Hey @kcq -- did the logs with --show-clogs output anything useful?
MShekow
commented
2 years ago Hi, I can confirm that docker-slim does not work in the GitLab SaaS runners, for the same reason as the OP indicated. See here. Pity.
kcq
commented
2 years ago @MShekow thanks a lot for providing the logs! Super helpful... and great that the app repo is available too https://gitlab.com/MShekow/docker-slim-example . We'll figure this out :)
MShekow
commented
2 years ago @kcq thanks for looking into it. The project (and docker-slim) does work in my organization's private GitLab instance, where we use self-hosted GitLab runners (with "docker" executor), which are configured to mount the host machine's Docker socket into the CI job containers (so we are not using Docker-in-Docker there, but socket mounting). This means that the problem somehow seems to be caused by using the DinD service.
kcq
commented
2 years ago @MShekow yes, mounting the docker socket with Gitlab's setup is not really an option. Need to use DOCKER_HOST instead. Double checking the exact configs...
kcq
commented
2 years ago Also looks like the main app is not picking up the right IP address when it's trying to talk to the temporary container
MShekow
commented
2 years ago Do you have any idea what causes this?
Nabsku
commented
6 months ago Is there anything I can do to help debug this? Currently facing this issue and don't want to mount sockets. Using DIND:Rootless.
kcq
commented
6 months ago Is there anything I can do to help debug this? Currently facing this issue and don't want to mount sockets. Using DIND:Rootless.
Not mounting the socket should be possible, but rootless hasn't been tested. I'd recommend trying a rootless setup locally if you want to double check if it works. The big question here is the use of fanotify and ptrace. Need to see if they'd work running rootless.
Nabsku
commented
6 months ago Is there anything I can do to help debug this? Currently facing this issue and don't want to mount sockets. Using DIND:Rootless.
Not mounting the socket should be possible, but rootless hasn't been tested. I'd recommend trying a rootless setup locally if you want to double check if it works. The big question here is the use of fanotify and ptrace. Need to see if they'd work running rootless.
Using sensor-ipc-mode proxy on dind:rootless I get this:
cmd=build info=container message='obtained IP address' ip='172.17.0.2' cmd=build info=cmd.startmonitor status='sent' cmd=build info=event.error status='received' data='SensorError{Op:sensor.fanotify.Run/fanapi.Initialize,Kind:call.error,Wrapped:{Type=syscall.Errno,Info=operation not permitted,Line:47,File:github.com/docker-slim/docker-slim/pkg/app/sensor/monitors/fanotify/monitor.go}}' cmd=build state=exited code=-124 component=container.inspector version=linux|Transformer|1.37.3-42-g768c393|768c3930f23c509b3f5b39da3940e7cc7a22efc3|2022-02-27_06:38:30PM
So I'd wager dind:rootless might be a no-go here.
Expected Behavior
docker-slim should run without error when run in a GitLab CI environment
Actual Behavior
docker-slim fails with timeout error.
Steps to Reproduce the Problem
.gitlab-ci.ymlfile:And here's the Dockerfile for the image being used in case that's helpful:
https://gitlab.com/megabyte-labs/dockerfile/ci-pipeline/dockerslim/-/blob/master/Dockerfile