Closed gireeshpunathil closed 4 years ago
@gireeshpunathil Thanks for the issue. Most of the team is enjoy PTO until the new year. If we don't get to it before, we'll definitely dive in at the start of the year.
@mheon FYI
👍 thanks @TomSweeneyRedHat, you have been greatly helpful as always! sure, let us tackle this in the new year!
Are you using a .dockerignore file?
I am not; but can try that, if required. Any steps / suggestions?
No we have seen performance issues when using .dockerignore
Could you try a newer version of buildah to see if it has an issue, say on a Fedora box?
@rhatdan - thnx. I don't have a fedora box, but created a container from fedora:latest
and tried, but got this error
[root@a099739f4c99 foo]# buildah version
Version: 1.12.0
Go Version: go1.13.5
Image Spec: 1.0.1-dev
Runtime Spec: 1.0.1-dev
CNI Spec: 0.4.0
libcni Version:
image Version: 5.0.0
Git Commit:
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
[root@a099739f4c99 foo]# uname -a
Linux a099739f4c99 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@a099739f4c99 foo]# cat /etc/fedora-release
Fedora release 31 (Thirty One)
[root@a099739f4c99 foo]# buildah bud .
ERRO 'overlay' is not supported over overlayfs
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver
[root@a099739f4c99 foo]#
any suggestions?
This looks like you have storage on an overlayfs file system? Are you running buildah inside of a container?
yes, in this case - as I don't have a normal fedora system.
Then volume mount a directory on /var/lib/containers and overlay will work.
@rhatdan - I am unable to upgrade buildah
, looks like this is the best version that is available for fedora. Am I using the right fedora
in the first place?
# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
# buildah version
Version: 1.9.0
Go Version: go1.10.8
Image Spec: 1.0.0
Runtime Spec: 1.0.0
CNI Spec: 0.4.0
libcni Version:
Git Commit:
Built: Wed Dec 31 16:00:00 1969
OS/Arch: linux/amd64
# yum upgrade buildah
Loaded plugins: product-id, search-disabled-repos
No packages marked for update
#
You are using RHEL.
Did you try to mount a file system at /var/lib/containers?
@rhatdan - thnx
You are using RHEL.
Yes, unfortunately I don't have system with fedora installed, I am using a container (fedora:latest) in a rhel host
Did you try to mount a file system at /var/lib/containers?
Yes, it did not make any visible difference w.r.t performance
Perhaps this will fix the issue. https://github.com/containers/buildah/pull/2070
thanks @rhatdan for fixing this. do you know when a release is going to be cut out of it? I tried building buildah
from source by following the doc, but got into issues:
# github.com/containers/buildah/vendor/github.com/vbatts/tar-split/archive/tar
vendor/github.com/vbatts/tar-split/archive/tar/writer.go:178:11: undefined: strings.Builder
# github.com/containers/buildah/vendor/github.com/vbauerster/mpb/v4/internal
vendor/github.com/vbauerster/mpb/v4/internal/percentage.go:14:9: undefined: math.Round
# github.com/containers/buildah/vendor/github.com/docker/docker/pkg/archive
vendor/github.com/docker/docker/pkg/archive/archive.go:365:5: hdr.Format undefined (type *tar.Header has no field or method Format)
vendor/github.com/docker/docker/pkg/archive/archive.go:365:15: undefined: tar.FormatPAX
vendor/github.com/docker/docker/pkg/archive/archive.go:1160:7: hdr.Format undefined (type *tar.Header has no field or method Format)
vendor/github.com/docker/docker/pkg/archive/archive.go:1160:17: undefined: tar.FormatPAX
vendor/github.com/docker/docker/pkg/archive/copy.go:346:7: hdr.Format undefined (type *tar.Header has no field or method Format)
vendor/github.com/docker/docker/pkg/archive/copy.go:346:17: undefined: tar.FormatPAX
[root@g1 buildah]#
@gireeshpunathil Thanks for the heads up on the install instructions, I'm not quite sure what's going on there. I'll most likely spin up a new release by early next week for Buildah.
@rhatdan @TomSweeneyRedHat - some updates.
when I upgraded the go version to 1.13.6 (by default, it was installing 1.9.4 for whatever reason), the build issues went away. And with that, I am able to custom build and test the patch for https://github.com/containers/buildah/pull/2070
[root]# du -ms .
41 .
[root]
old case
[root]# buildah version
Version: 1.9.0
Go Version: go1.10.8
Image Spec: 1.0.0
Runtime Spec: 1.0.0
CNI Spec: 0.4.0
libcni Version:
Git Commit:
Built: Wed Dec 31 16:00:00 1969
OS/Arch: linux/amd64
[root]#
[root]# time buildah bud .
STEP 1: FROM scratch
STEP 2: COPY . /project
STEP 3: COMMIT
Getting image source signatures
Copying blob fd191a377db7 done
Copying config 4511d8faeb done
Writing manifest to image destination
Storing signatures
4511d8faeb8d608bbdcfeecb6fcb4570b0b33e57ad9b031fb1e323515cbeaf49
real 1m27.032s
user 0m56.247s
sys 0m34.159s
good case
[root]# ./buildah version
Version: 1.14.0-dev
Go Version: go1.13.6
Image Spec: 1.0.1-dev
Runtime Spec: 1.0.1-dev
CNI Spec: 0.4.0
libcni Version: v0.7.1
image Version: 5.1.0
Git Commit: 4e23b7a
Built: Sat Jan 11 03:30:08 2020
OS/Arch: linux/amd64
[root]# time ./buildah bud .
STEP 1: FROM scratch
STEP 2: COPY . /project
STEP 3: COMMIT
Getting image source signatures
Copying blob 83a7eb8c927e done
Copying config f549711ba9 done
Writing manifest to image destination
Storing signatures
f549711ba95d5fe7b911d36bcfef16940f4f49fbaf62e6cc3f7da1d411832985
f549711ba95d5fe7b911d36bcfef16940f4f49fbaf62e6cc3f7da1d411832985
real 0m5.305s
user 0m3.652s
sys 0m2.383s
as we can see, the 1.14.0-dev
version just takes 5 seconds, as opposed to the old one (87 seconds).
thanks once again, will wait upon quay.io for a release!
Glad it's working and thanks for the Go version update too. That's not something that I've run into. I can't take much credit on the fix though, @nalind did all the heavy lifting on this one.
thanks @nalind for quickly turning around this!
@gireeshpunathil - Could you please accommodate Kyle G. Christianson request: Please make sure to test this the node pipelines. This was another reason we stayed on an old version of buildah: https://github.com/appsody/appsody-buildah/issues/10. Thank you.
@marikaj123 - looks like you probably meant to put this comment in https://github.com/kabanero-io/kabanero-pipelines/issues/121 instead of here? either way, answer is yes, will do.
As this appears to be addressed in Buildah based on @gireeshpunathil 's comments, I'm closing this. @gireeshpunathil Please feel free to open another issue if things aren't going as expected with Buildah.
Description
I am composing an image from a large (~200MB) project structure using
buildah bud
that never completes. Here is the full bug report for reference: https://github.com/kabanero-io/kabanero-pipelines/issues/121I have brought it down to a simple recreate for easy diagnosis
Steps to reproduce the issue:
$ ls -lrt
$ cat Dockerfile
$ cat foo.sh
As you can see, a folder structure with 1K+ directories inside an 4MB of data takes 1 minute + where as a a monolith file of 5MB just takes over a second.
Describe the results you received:
unexpected delay in
COPY
directive. When I tried to trace usingstrace
I believe things started moving faster, but because of the change in timings that is involved withstrace
, results are never conclusiveDescribe the results you expected:
Few seconds for copying - parity with
docker build
commandOutput of
rpm -q buildah
orapt list buildah
:Output of
buildah version
:Output of
podman version
if reporting apodman build
issue:*Output of `cat /etc/release`:**
Output of
uname -a
:Output of
cat /etc/containers/storage.conf
: