Closed adrian-moisa closed 6 months ago
Do you have a big "build context" (the one you pass as the last argument), that needs tarring up before building?
mkdir empty
cp Dockerfile empty
podman build -t test -f Dockerfile empty
https://docs.docker.com/build/building/context/#dockerignore-files
TLDR - podman runs slow because it was building from the root of the monorepo. Not all heavy folders where ignored in the dockerignore. used ncdu to find the heavy ones. Even with ignore it still has some lag but it's way better. Still it has best performance when building at individual project folder level instead of monorepo root. Docker can build at monorepo without this delay at all
Great guess. Indeed, I just did the test, and in an isolated folder the build runs as expected. Near instant for a "FROM alpine".
However there's a catch for me. I'm working in a not so large, not so small monorepo. I decided to plan all my builds from the root level of the repo because it's still "work in progress". Most of the go apis borrow models from each other. The shared models will be published to a dedicated artifactory and imported as a dependency. Thus I could copy only each individual api to it's own dockerfile. it's going to be a while until I do this refactor. Until then I need to build from root.
I used ncdu
to analyse the folders size. The major culprit seems to be the Flutter app folder, especially the build
(2GB) and .dart_tool
(800MB) folders.
This also explains why the test on the other mac was even worse. Because I tried it on the desktop and there my gf (a non-tech person) stores a lot of files on the desktop. Which means that context was even bigger than my monorepo. Initially I wasn't aware that her desktop is so full of subfolders.
I tried an improved .dockerignore
and it does the trick. It's far faster than no .dockerignore
for the build
and .dart_tool
folders. However I still notice a significant 10-20s of intensive scanning until it figures out what to ignore. With or without logging the ignored files it's still quite some time. Compared with 2-3 minutes prior it's way better. But compared to docker it's way slower. Docker can pull it off in 3-4 seconds from monorepo root. I can also see docker does not do any TCP traffic (no VM i guess, idk).
It seems that #21627 #20965 both talk about the same issue. I don't recall stumbling on them. Maybe I did but skimmed. Can't say for sure. Tbh also google could do a better job maybe.
Suggestions for improvement: Better logs I spent 3 days trying to debug this thing. I rarely give up on my debugs. Yesterday I was ready to completely switch to docker. I would describe myself as an expert in devops but I'm neither a junior. This problem got the best of me. I never thought of changing the build context. Again could be due to the fact that I only have 6 months of running stuff in containers. So my argument is that despite my past experiences I did not find enough clues in the logs to understand that the build context is the problem.
--log-level=debug
. Therefore there was no clue given from the debug logs of what goes inside of the "black box". I had no way to guess that the large build folder in the flutter app was slowing everything down..dockerignore
could greatly influence the performance of podman builds. I was thinking of it more of a way to keep things tidy. Again, maybe a log message could hint at this fact.Side question - Why still stick with podman instead of docker? This entire experience made me question deeply my choice of using podman. This is not my first fight with podman.
Because this is my first time in the world of containers I chose podman mostly because of the heavy marketing about rootless and of the docker desktop license. Now looking back at the history of choices I wonder if the docker license is really such a big deal. Apparently docker also offers rootless. And anyway, when I deploy my Kubernetes cluster it wont use podman. So basically right now, after all these battle scars I have to question deeply my allegiance to podman. Can you provide a strong reason where podman is still the winner?
Thanks for the support!
Speeding up podman build
by using .containerignore
or .dockerignore
could be a useful performance tip for
https://github.com/containers/podman/blob/main/docs/tutorials/performance.md
I started writing such a tip: https://github.com/eriksjolund/podman/blob/add-performance-tip-containerignore/docs/tutorials/performance.md
Because this is my first time in the world of containers I chose podman mostly because of the heavy marketing about rootless and of the docker desktop license. Now looking back at the history of choices I wonder if the docker license is really such a big deal. Apparently docker also offers rootless. And anyway, when I deploy my Kubernetes cluster it wont use podman. So basically right now, after all these battle scars I have to question deeply my allegiance to podman. Can you provide a strong reason where podman is still the winner?
@adrian-moisa i read your write-ups. Apparently you had quite an experience. You should feel free to choose your container solution as you see fit; upstream Podman is largely development oriented and as such, you retain the choice. It is no secret that we have had several problems with our Mac implementation; some of which were well out of our control and some of which were bugs, etc. As you mention, or at least I assume you are referring to, upstream QEMU and then subsequently brew migrated to a newer QEMU where the firmware required for aarch64 was missing (or no present ... i forget frankly). I would say that is out of our hands. As for minikube, that is obviously a different project as well.
I can say that the future for AppleSilicon Macs looks bright for podman. We are currently wrapping up development for Podman 5, which no longer uses QEMU and will use Apple's native hypervisor. As such, we are also able to take advantage of virtiofs shares which is significantly more performant than QEMU and the 9P implementation.
As this issue stands, I'm not sure what you are asking of the Podman community and as such, it could theoreticaly get closed. I would recommend altering the title of the issue to reflect what you think is wrong or questionable. Also, you took the time to write this issue; perhaps you can take the next step and submit documentation or code that addresses problems? We love community submissions.
Whatever you choose, good luck with your container adventures!
A friendly reminder that this issue had no activity for 30 days.
I am close this as dup of https://github.com/containers/podman/issues/21627
Also there have been quite a few contributions recently to make sending/extracting the build tarball faster
Issue Description
Mac M1, Podman build spends a lot of time doing "something" not sure what before the actual build steps. Lots of network usage via gvproxy. Even for a tiny image such as alpine. gvproxy shows at least 1GB of trafic for each of these attempts.
I can confirm that this was not a problem in for the entire of February. I did many builds and they were going quite fast, epsecially when using a multistage build whith preloaded module dependencies. It took less than 30s to complete such a build.
At some point I had a walk outside, the mac went into sleep mode and when I returned I found some unusual behaviour in minikube. I thought that maybe I need to reinstall everything. Since the reinstall, all my builds have this minimum 3-4 minutes of hang time.
I tried:
--log-level=debug
. There's no printout during this long period of high intensity gvproxy traffic.podman machine init --image-path https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231204.3.3/aarch64/fedora-coreos-39.20231204.3.3-qemu.aarch64.qcow2.xz
podman machine ssh
and then tried to see intop
who uses the CPU. Found that openssh is buys. I couldn't figure out what commands is executing:Tried various ways to figure out who is process/command doing such large traffic before the actual build. No luck
Full uninstall procedure Eventually I figured out that all I need to remove is minikube. That's why I sorted by categories the cleanup procedure.
Restart procedure Note: You have to prepare your .rb files as instructed in the tutorial if you want to rollback.
Steps to reproduce the issue
podman build -t test -f Dockerfile .
Describe the results you received
Describe the results you expected
podman should be working fast right away
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
No response
Additional information
No response