search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.41k stars 2.31k forks source link

podman build no longer works in 1.7.0 #4905

Closed lfarkas closed 4 years ago

lfarkas commented 4 years ago

with the latest fedora podman-1.7.0-2.fc31.x86_64 update podman build no longer works ie, it's hang in a copy command:

STEP 6: WORKDIR /var/www/html
d419d56d6e07a78728c2b00977fd0f01f0ceec2b5d9a9afcf24fcccef2dfb7d0
STEP 7: COPY $BUILD_DST .

forever. it's start to get really annoying that none of the latest fedora podman build working (the previous totally ignore .dockerignore which result a catastrophic images).

is there any way to debug what's happening or in what it's hang?

mheon commented 4 years ago

Adding --log-level=debug to the command line might help.

@TomSweeneyRedHat PTAL

giuseppe commented 4 years ago

do we have https://github.com/containers/buildah/pull/2072 in the vendored copy?

lfarkas commented 4 years ago

ok it's build, but the build takes about a halfen hour while it was about a minutes in the previous version. from the --log-level=debug the build is full of (for all file): DEBU[0030] error closing backward: invalid argument

rhatdan commented 4 years ago

The Buildah copy fix will be in 1.7.1, but we should get this out ASAP. @mheon can we cut a new release?

mheon commented 4 years ago

Uh. Let's give it a few days and do it while we have everyone in the Brno office.

TomSweeneyRedHat commented 4 years ago

@rhatdan do you suspect this was cured by the Overlayv2 fix in c/storage that went into Buildah this weekend? @lfarkas, if so, any chance you could try upstream Buildah?

lfarkas commented 4 years ago

i can try anything if you give me an rpm/binary and or command to test

vrothberg commented 4 years ago

@lfarkas could you share your Dockerfile as a reproducer? This would simplify checking if your issue has been solved.

lfarkas commented 4 years ago

the dockerfile itself is very simple or actually large but everything run fast but the relevant part is only one line: COPY . . which try to copy the generated laravel/php code into the image and during the copy it's full of errors like this:

DEBU[0002] copyFileWithTar(/home/lfarkas/work/develnet/hostop/hostop/webadmin/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/css/font-awesome.css, /home/lfarkas/.local/share/containers/storage/overlay/23455afa8df20434492a450d7fdc42d447735c4ce6c92cfdcbc5fb6b63687712/merged/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/css/font-awesome.css) 
DEBU[0002] error closing font-awesome.css: invalid argument 
DEBU[0002] copyFileWithTar(/home/lfarkas/work/develnet/hostop/hostop/webadmin/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/css/font-awesome.min.css, /home/lfarkas/.local/share/containers/storage/overlay/23455afa8df20434492a450d7fdc42d447735c4ce6c92cfdcbc5fb6b63687712/merged/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/css/font-awesome.min.css) 
DEBU[0002] error closing font-awesome.min.css: invalid argument 
DEBU[0002] copyFileWithTar(/home/lfarkas/work/develnet/hostop/hostop/webadmin/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts, /home/lfarkas/.local/share/containers/storage/overlay/23455afa8df20434492a450d7fdc42d447735c4ce6c92cfdcbc5fb6b63687712/merged/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts) 
DEBU[0002] copyFileWithTar(/home/lfarkas/work/develnet/hostop/hostop/webadmin/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts/FontAwesome.otf, /home/lfarkas/.local/share/containers/storage/overlay/23455afa8df20434492a450d7fdc42d447735c4ce6c92cfdcbc5fb6b63687712/merged/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts/FontAwesome.otf) 
DEBU[0002] error closing FontAwesome.otf: invalid argument 
DEBU[0002] copyFileWithTar(/home/lfarkas/work/develnet/hostop/hostop/webadmin/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts/fontawesome-webfont.eot, /home/lfarkas/.local/share/containers/storage/overlay/23455afa8df20434492a450d7fdc42d447735c4ce6c92cfdcbc5fb6b63687712/merged/public/vendor/backpack/bootstrap-iconpicker/icon-fonts/font-awesome-4.3.0/fonts/fontawesome-webfont.eot) 
DEBU[0002] error closing fontawesome-webfont.eot: invalid argument

and it takes very long time. since there are 26180 files in the directory. if i leave it there it can finish...once:-) real 28m9.044s user 18m19.830s sys 16m43.759s and almost all time spend in the copy command.

if you need a real example just use this demo: https://github.com/Laravel-Backpack/Demo

vrothberg commented 4 years ago

@TomSweeneyRedHat @rhatdan, do we need to do the vendor dance before or are we good to test with the latest master?

mheon commented 4 years ago

I don't think we have an updated Buildah landed.

TomSweeneyRedHat commented 4 years ago

If the c/storage fix fixed the issue, we won't have that until Buildah 1.13.2+ lands. That's a WIP atm. So we are still doing the vendor dance here.

rhatdan commented 4 years ago

We have issues with buildah vendoring into Podman, due to the containers.conf merger into Buildah. I am working on reverting/fixing some issues on this now, Hopefully this will get merged fairly quickly. But I would rather not rush containers.conf buildah out right now.

ErichDonGubler commented 4 years ago

I was able to work around this issue in Fedora 31 by installing the buildah package, which as I understand it is then used by an installed podman package.

rhatdan commented 4 years ago

It is not really used, it is more embedded. Basically Podman uses the buildah code like a shared library inside of itself, it does NOT execute the Buildah command or need Buildah installed.

lfarkas commented 4 years ago

it's still not working (ie very slow) with 1.8.0

TomSweeneyRedHat commented 4 years ago

@nalind FYI

rhatdan commented 4 years ago

Since this is a well known buildah issue, I am going to close this and hope that this gets fixed in Buildah soon and then Podman gets the update.

lfarkas commented 4 years ago

it's nice that you close it but after 2 month it's still not fixed and not working on fedora:-(

rhatdan commented 4 years ago

@lfarkas If this is not working in Buildah please open a bug over there.

rhatdan commented 4 years ago

There is an ongoing rewrite of copy in buildah going on. We have lots of issues/complaints about copy being too slow. We have fixed a few, but @nalind Thinks that a full rewrite is necessary and he has been working on it. Hopefully we will have something soon.

lfarkas commented 4 years ago

@lfarkas If this is not working in Buildah please open a bug over there.

it's not working in podman build (since most people use it). what's a it's a regression since it was working in 1.4.4 el7!

lfarkas commented 4 years ago

what's more even if the v1.8.2 release notes said: Updated Buildah to v1.14.3 it's still not working.

mheon commented 4 years ago

Do we have an issue about this on the Buildah side? @rhatdan @TomSweeneyRedHat

If not we should get one written

TomSweeneyRedHat commented 4 years ago

I'm not sure if we have a particular issue in the Buildah issues, but it's a problem that's also been reported via Bugzilla and @nalind has been diligently working on it.

mheon commented 4 years ago

Basically, I want a tracker issue that we can point people to, that will be closed when this is actually done and released. If we have a public BZ about that, that ought to be fine.

TomSweeneyRedHat commented 4 years ago

The BZ is at: https://bugzilla.redhat.com/show_bug.cgi?id=1756986 which is based off of https://github.com/containers/buildah/issues/1714

lfarkas commented 4 years ago

The BZ is at: https://bugzilla.redhat.com/show_bug.cgi?id=1756986 which is based off of containers/buildah#1714

which is totally false! since it's was closed as "Fixed in buildah-1.11.2-2" but still not working with buildah-1.14.3-1.fc31.x86_64

TomSweeneyRedHat commented 4 years ago

@nalind PTAL

lfarkas commented 3 years ago

it's still not fixed and still not working in podman-2.0.2-1.fc32.x86_64. why did you close it???

mheon commented 3 years ago

If it's not working, please open a fresh issue against Buildah, given this is a bug in Buildah code.

lfarkas commented 3 years ago

ok i know that all go code is statically linked. and podman is bad because of buildah, BUT still it's a podman bug! if I try to build with podman it's not working. and I do NOT really case about when will be it fixed in buildah. since even if it's fixed in buildah and do not fixed in podman then it's still a bug in podman. so please reopen this bug against podman until podman use such a version of buildah which fix this bug. what's more it's a regression since it was working in a earlier version and it's no longer works!

imho it's a same bug if I a program link against a wrong version of openssl and it's not working properly it's still bug in the given program.

rhatdan commented 3 years ago

@lfarkas At this point I am confused about what this issue is? If it is Podman build is slow when you use COPY command have a huge amount of files and are using a .dockerignore then that is fine, and is being worked on. That has existed since the beginning of Buildah and Podman, so I don't see this as a regression. If you are talking about something else, then perhaps we need a new issue. Bottom line is that having an open issue here on podman build is not going to be effective. Opening the issue in buildah bud would lead to a fix and then the fix will get vendored into Podman.

mheon commented 3 years ago

@lfarkas We use Buildah code for Podman build, but most Podman developers are not Buildah developers; if you file the bug here, all we'll do is ask the Buildah developers to look at it, and then they will probably transfer the bug over to their repo. Creating it there skips a great many steps and will make the process faster and easier for everyone.

The exception would be if you confirm that it is fixed with the latest buildah but is still broken on Podman. In that case, it's definitely our problem, maybe a too-old version? At this point, I don't think we've confirmed that the bug is actually present/not present in Buildah.