ksingh7
commented
2 years ago @kcq Hey Kyle, can you help with this, i am trying to minify a 5GB image (dockerfile) using docker-slim , but it breaks.
May be if you have time, you can give it a try
Open ksingh7 opened 2 years ago
ksingh7
commented
2 years ago @kcq Hey Kyle, can you help with this, i am trying to minify a 5GB image (dockerfile) using docker-slim , but it breaks.
May be if you have time, you can give it a try
kcq
commented
2 years ago @ksingh7 let me take a look (will try to repro in the am)
also wonder if you can add the --debug and --show-clogs flags to get more details (docker-slim --debug build rembg --http-probe=false --show-clogs)
eduardosramos
commented
2 years ago Problema comum de praticamente todos aqui: Reduz a imagem e depois não funciona. Me parece uma perda de tempo.
kcq
commented
2 years ago Problema comum de praticamente todos aqui: Reduz a imagem e depois não funciona. Me parece uma perda de tempo.
Naturally it'll be common in bug reports... That's the whole point of reporting bugs. Nobody opens issues just to say they successfully minified their image :-) In this case it's a memory use issue in the sensor. Either way, some apps do require additional configs and it does take a few tries to arrive at the right combination of flags
decarboxy
commented
2 years ago Hey there, I had exactly the same error with a large (30GB) image. I ran it with the --debug flag, the results of which are below, if they're of any help.
you mentioned it was a memory use issue, which i guess makes sense given the size of the container. I know i'm not running out of memory on the machine itself.
cmd=build state=container.inspection.finishing
time="2022-04-22T21:32:58Z" level=debug msg="RunContainer: Docker event monitor stopped" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:32:58Z" level=debug msg="ipc.Client.SendCommand() cmd channel call data='{\"name\":\"cmd.monitor.stop\"}'\n"
time="2022-04-22T21:32:58Z" level=debug msg="channel.CommandClient.Call: calling c.Write()"
time="2022-04-22T21:32:58Z" level=debug msg="channel.Client.Write: remote=127.0.0.1:32779 -> local=127.0.0.1:38762 - conn.Write wc=110 err=<nil> [time=1650663178591688638]"
time="2022-04-22T21:32:58Z" level=debug msg="channel.CommandClient.Call: calling c.Read()"
time="2022-04-22T21:32:58Z" level=debug msg="channel.Client.Read() - (0xc0005c90e0)reader.ReadString => err=<nil> raw=\"[<|]{\\\"tid\\\":\\\"1650663178591618800.c64981855ad8681d\\\",\\\"type\\\":\\\"ft.response\\\",\\\"body\\\":{\\\"status\\\":\\\"ok\\\"}}[|>]\\n\""
time="2022-04-22T21:32:58Z" level=debug msg="channel.Client.Read: got raw frame ='[<|]{\"tid\":\"1650663178591618800.c64981855ad8681d\",\"type\":\"ft.response\",\"body\":{\"status\":\"ok\"}}[|>]\n'"
time="2022-04-22T21:32:58Z" level=debug msg="channel.Client.Read: frame data => tid=1650663178591618800.c64981855ad8681d type=ft.response body='{\"status\":\"ok\"}'"
time="2022-04-22T21:32:58Z" level=debug msg="channel.CommandClient.Call: c.Read() (tid=1650663178591618800.c64981855ad8681d type=ft.response) result.data='{\"status\":\"ok\"}'"
time="2022-04-22T21:32:58Z" level=debug msg="ipc.Client.SendCommand() cmd channel call response='{\"status\":\"ok\"}'\n"
time="2022-04-22T21:32:58Z" level=debug msg="'stop' monitor response => '&{ok}'" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:32:58Z" level=info msg="waiting for the container to finish its work..." app=docker-slim command=build component=container.inspector
time="2022-04-22T21:33:26Z" level=debug msg="channel.Client.Read() - (0xc0005c9200)reader.ReadString => err=EOF raw=\"\""
time="2022-04-22T21:33:26Z" level=debug msg="channel.Client.Read: connection done..."
time="2022-04-22T21:33:26Z" level=error msg="ipc.Client.GetEvent(): event channel error = EOF\n"
time="2022-04-22T21:33:26Z" level=debug msg="sensor event => '<nil>'" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:33:26Z" level=warning msg="docker-slim: warning" error=EOF stack="goroutine 1 [running]:\nruntime/debug.Stack(0xc00031d570, 0x49f472, 0xc00012e0c0)\n\truntime/debug/stack.go:24 +0x9f\ngithub.com/docker-slim/docker-slim/pkg/util/errutil.WarnOn(0x10cb3c0, 0xc000116030)\n\tgithub.com/docker-slim/docker-slim/pkg/util/errutil/errutil.go:38 +0x54\ngithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container.(*Inspector).FinishMonitoring(0xc000491600)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container/container_inspector.go:1287 +0x3b1\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.OnCommand(0xc00013f260, 0xc0003e8300, 0x7ffc2223802c, 0x59, 0x0, 0x0, 0x1748a78, 0x0, 0x0, 0x0, ...)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/handler.go:1304 +0x449d\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.glob..func1(0xc00055a800, 0x86, 0x130)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/cli.go:662 +0x3688\ngithub.com/urfave/cli/v2.(*Command).Run(0x16dee00, 0xc000522580, 0x0, 0x0)\n\tgithub.com/urfave/cli/v2@v2.3.0/command.go:163 +0x4ed\ngithub.com/urfave/cli/v2.(*App).RunContext(0xc000134000, 0x10e2d20, 0xc00012a010, 0xc000132000, 0xc, 0xc, 0x0, 0x0)\n\tgithub.com/urfave/cli/v2@v2.3.0/app.go:313 +0x81f\ngithub.com/urfave/cli/v2.(*App).Run(...)\n\tgithub.com/urfave/cli/v2@v2.3.0/app.go:224\ngithub.com/docker-slim/docker-slim/pkg/app/master.Run()\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/app.go:15 +0x6c\nmain.main()\n\tgithub.com/docker-slim/docker-slim/cmd/docker-slim/main.go:15 +0x54\n" version="linux|Transformer|1.37.5|86fbd29ab3549fa564e87e4770178480cb0542d3|2022-03-21_06:10:20AM"
time="2022-04-22T21:33:26Z" level=debug msg="sensor event => '<nil>'" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:33:26Z" level=debug msg="ipc.Client.SendCommand() cmd channel call data='{\"name\":\"cmd.sensor.shutdown\"}'\n"
time="2022-04-22T21:33:26Z" level=debug msg="channel.CommandClient.Call: calling c.Write()"
time="2022-04-22T21:33:26Z" level=debug msg="channel.Client.Write: remote=127.0.0.1:32779 -> local=127.0.0.1:38762 - conn.Write wc=113 err=<nil> [time=1650663206288823211]"
time="2022-04-22T21:33:26Z" level=debug msg="channel.CommandClient.Call: calling c.Read()"
time="2022-04-22T21:33:26Z" level=debug msg="channel.Client.Read() - (0xc0005c90e0)reader.ReadString => err=EOF raw=\"\""
time="2022-04-22T21:33:26Z" level=debug msg="channel.Client.Read: connection done..."
time="2022-04-22T21:33:26Z" level=error msg="channel.CommandClient.Call: c.Read error = EOF"
time="2022-04-22T21:33:26Z" level=error msg="ipc.Client.SendCommand() cmd channel call error=EOF\n"
time="2022-04-22T21:33:26Z" level=debug msg="error sending 'shutdown' => 'EOF'" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:33:26Z" level=debug msg="'shutdown' sensor response => '<nil>'" app=docker-slim command=build component=container.inspector
time="2022-04-22T21:33:26Z" level=info msg="shutting down 'fat' container..." app=docker-slim command=build
time="2022-04-22T21:33:26Z" level=error msg="dockerutil.CopyFromContainer: dclient.DownloadFromContainer() error = API error (404): Could not find the file /opt/dockerslim/artifacts/creport.json in container 76a81c862c47b72897aef67a278f3cc0a4eff25c338e54aa6bf2d4b8766e5c84"
time="2022-04-22T21:33:26Z" level=fatal msg="docker-slim: failure" error="API error (404): Could not find the file /opt/dockerslim/artifacts/creport.json in container 76a81c862c47b72897aef67a278f3cc0a4eff25c338e54aa6bf2d4b8766e5c84" stack="goroutine 1 [running]:\nruntime/debug.Stack(0xc000046000, 0xc00031d6b8, 0x419c85)\n\truntime/debug/stack.go:24 +0x9f\ngithub.com/docker-slim/docker-slim/pkg/util/errutil.FailOn(0x10cb7e0, 0xc0005cb760)\n\tgithub.com/docker-slim/docker-slim/pkg/util/errutil/errutil.go:28 +0x54\ngithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container.(*Inspector).ShutdownContainer(0xc000491600, 0x4, 0xc00031e2a8)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container/container_inspector.go:1198 +0x648\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.OnCommand(0xc00013f260, 0xc0003e8300, 0x7ffc2223802c, 0x59, 0x0, 0x0, 0x1748a78, 0x0, 0x0, 0x0, ...)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/handler.go:1307 +0x450f\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.glob..func1(0xc00055a800, 0x86, 0x130)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/cli.go:662 +0x3688\ngithub.com/urfave/cli/v2.(*Command).Run(0x16dee00, 0xc000522580, 0x0, 0x0)\n\tgithub.com/urfave/cli/v2@v2.3.0/command.go:163 +0x4ed\ngithub.com/urfave/cli/v2.(*App).RunContext(0xc000134000, 0x10e2d20, 0xc00012a010, 0xc000132000, 0xc, 0xc, 0x0, 0x0)\n\tgithub.com/urfave/cli/v2@v2.3.0/app.go:313 +0x81f\ngithub.com/urfave/cli/v2.(*App).Run(...)\n\tgithub.com/urfave/cli/v2@v2.3.0/app.go:224\ngithub.com/docker-slim/docker-slim/pkg/app/master.Run()\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/app.go:15 +0x6c\nmain.main()\n\tgithub.com/docker-slim/docker-slim/cmd/docker-slim/main.go:15 +0x54\n" version="linux|Transformer|1.37.5|86fbd29ab3549fa564e87e4770178480cb0542d3|2022-03-21_06:10:20AM"Hey there, I had exactly the same error with a large (30GB) image. I ran it with the
--debugflag, the results of which are below, if they're of any help.
Hey Sam :-) Can you also add the --show-clogs command level param. It'll help understand what happened with the sensor in the container. The creport.json should be there unless the sensor crashed :)
decarboxy
commented
2 years ago I ran in debug mode with --show-clogs, the complete logs are here: slim-logs.txt
the container is an open source project, the dockerfile is here: https://github.com/CyrusBiotechnology/RoseTTAFold/blob/main/Dockerfile, i can provide a command line and reasonable minimal input if it would be useful to you.
I did notice that after the application in the container completed successfully the sensors were using a lot of cpu, as was dockershim, for about 40 minutes. Is it possible that due to the container things have gotten "behind" in some way and i should wait for the sensors to become idle before pressing enter?
sureshkachwa
commented
1 year ago I have the below error when trying to slim the image, the actual image size is 4.24GB.
Below are commands I used to build and slim the images respectively
docker build --network=host -t moon:${VERSION}
docker-slim build --target moon:${VERSION} --tag moon:${VERSION} --http-probe=false --include-path /
Error below:
cmd=build state=container.inspection.finishing
time="2023-05-10T15:16:19+05:30" level=error msg="ipc.Client.GetEvent(): event channel error = EOF\n"
time="2023-05-10T15:16:19+05:30" level=warning msg="docker-slim: warning" error=EOF stack="goroutine 1 [running]:\nruntime/debug.Stack(0x0, 0xc0005962e0, 0x533665)\n\truntime/debug/stack.go:24 +0x9f\ngithub.com/docker-slim/docker-slim/pkg/util/errutil.WarnOn(0xfca5e0, 0xc000072040)\n\tgithub.com/docker-slim/docker-slim/pkg/util/errutil/errutil.go:38 +0x54\ngithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container.(*Inspector).FinishMonitoring(0xc0004abb00)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container/container_inspector.go:1104 +0x271\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.OnCommand(0xc0004f6960, 0xc0004c74f0, 0x7fffb8799b44, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/handler.go:1079 +0x409d\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.glob..func1(0xc00047b4a0, 0x0, 0xc0004b36f0)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/cli.go:629 +0x321b\ngithub.com/urfave/cli.HandleAction(0xd2e100, 0xeca9b0, 0xc00047b4a0, 0xc00047b4a0, 0x0)\n\tgithub.com/urfave/cli@v1.22.4/app.go:526 +0xfd\ngithub.com/urfave/cli.Command.Run(0xe824d9, 0x5, 0x0, 0x0, 0x1564aa0, 0x1, 0x1, 0xebe395, 0x6c, 0x0, ...)\n\tgithub.com/urfave/cli@v1.22.4/command.go:174 +0x58e\ngithub.com/urfave/cli.(*App).Run(0xc0003656c0, 0xc000030090, 0x9, 0x9, 0x0, 0x0)\n\tgithub.com/urfave/cli@v1.22.4/app.go:279 +0x7e8\ngithub.com/docker-slim/docker-slim/pkg/app/master.Run()\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/app.go:15 +0x54\nmain.main()\n\tgithub.com/docker-slim/docker-slim/cmd/docker-slim/main.go:8 +0x25\n" version="linux|Transformer|1.37.3|97eb3c8c7f8c4e295a26c13882feb3f0539ab50f|2021-12-10_09:37:29AM"
time="2023-05-10T15:16:19+05:30" level=error msg="channel.CommandClient.Call: c.Read error = EOF"
time="2023-05-10T15:16:19+05:30" level=error msg="ipc.Client.SendCommand() cmd channel call error=EOF\n"
time="2023-05-10T15:16:19+05:30" level=error msg="dockerutil.CopyFromContainer: dclient.DownloadFromContainer() error = API error (404): Could not find the file /opt/dockerslim/artifacts/creport.json in container 08dd4c141ff290a918cd791874f796cb58683e3c207d25242995bea860c9eaad"
time="2023-05-10T15:16:19+05:30" level=fatal msg="docker-slim: failure" error="API error (404): Could not find the file /opt/dockerslim/artifacts/creport.json in container 08dd4c141ff290a918cd791874f796cb58683e3c207d25242995bea860c9eaad" stack="goroutine 1 [running]:\nruntime/debug.Stack(0xc000050800, 0xc000596260, 0x419c85)\n\truntime/debug/stack.go:24 +0x9f\ngithub.com/docker-slim/docker-slim/pkg/util/errutil.FailOn(0xfca900, 0xc0005025c0)\n\tgithub.com/docker-slim/docker-slim/pkg/util/errutil/errutil.go:28 +0x54\ngithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container.(*Inspector).ShutdownContainer(0xc0004abb00, 0x4, 0xc000596e00)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/inspectors/container/container_inspector.go:1023 +0x65a\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.OnCommand(0xc0004f6960, 0xc0004c74f0, 0x7fffb8799b44, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/handler.go:1082 +0x410f\ngithub.com/docker-slim/docker-slim/pkg/app/master/commands/build.glob..func1(0xc00047b4a0, 0x0, 0xc0004b36f0)\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/commands/build/cli.go:629 +0x321b\ngithub.com/urfave/cli.HandleAction(0xd2e100, 0xeca9b0, 0xc00047b4a0, 0xc00047b4a0, 0x0)\n\tgithub.com/urfave/cli@v1.22.4/app.go:526 +0xfd\ngithub.com/urfave/cli.Command.Run(0xe824d9, 0x5, 0x0, 0x0, 0x1564aa0, 0x1, 0x1, 0xebe395, 0x6c, 0x0, ...)\n\tgithub.com/urfave/cli@v1.22.4/command.go:174 +0x58e\ngithub.com/urfave/cli.(*App).Run(0xc0003656c0, 0xc000030090, 0x9, 0x9, 0x0, 0x0)\n\tgithub.com/urfave/cli@v1.22.4/app.go:279 +0x7e8\ngithub.com/docker-slim/docker-slim/pkg/app/master.Run()\n\tgithub.com/docker-slim/docker-slim/pkg/app/master/app.go:15 +0x54\nmain.main()\n\tgithub.com/docker-slim/docker-slim/cmd/docker-slim/main.go:8 +0x25\n" version="linux|Transformer|1.37.3|97eb3c8c7f8c4e295a26c13882feb3f0539ab50f|2021-12-10_09:37:29AM" ```
Expected Behavior
Actual Behavior
Steps to Reproduce the Problem
1. Grab this docker file, from [here](https://github.com/ksingh7/rembg/blob/main/Dockerfile) 2. Build docker image (standard) `docker build -t rembg` 3. This will produce a 5.0 GB rembd docker image 4. Invoke docker-slim to minify this image `docker-slim build rembg --http-probe=false` --- Specifications ================= - Version: 1.37.3 - Platform: MacOS - Darwin --- Logs ===== ``` $ docker-slim build rembg --http-probe=false docker-slim: message='join the Gitter channel to ask questions or to share your feedback' info='https://gitter.im/docker-slim/community' docker-slim: message='join the Discord server to ask questions or to share your feedback' info='https://discord.gg/9tDyxYS' docker-slim: message='Github discussions' info='https://github.com/docker-slim/docker-slim/discussions' cmd=build info=exec message='changing continue-after from probe to nothing because http-probe is disabled' cmd=build info=exec message='changing continue-after to enter' cmd=build state=started cmd=build info=params target.type='image' target='rembg' continue.mode='enter' rt.as.user='true' keep.perms='true' tags='' cmd=build state=image.inspection.start cmd=build info=image size.human='5.1 GB' id='sha256:ebf916f1382706af6dc7f172e4dd10db8e53c4016accdfe759e678a44a6c9951' size.bytes='5086630999' cmd=build info=image.stack index='0' name='rembg:latest' id='sha256:ebf916f1382706af6dc7f172e4dd10db8e53c4016accdfe759e678a44a6c9951' cmd=build state=image.inspection.done cmd=build state=container.inspection.start cmd=build info=container status='created' name='dockerslimk_24613_20220222043146' id='c6558fa8fa08e1ccbfd7c28bd23da098788d369fb42e9e57d991bf1bafb1d5eb' cmd=build info=cmd.startmonitor status='sent' cmd=build info=event.startmonitor.done status='received' cmd=build info=container id='c6558fa8fa08e1ccbfd7c28bd23da098788d369fb42e9e57d991bf1bafb1d5eb' target.port.list='' target.port.info='' message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER' name='dockerslimk_24613_20220222043146' cmd=build info=continue.after mode='enter' message='provide the expected input to allow the container inspector to continue its execution' cmd=build prompt='USER INPUT REQUIRED, PRESS