Bug: performance problem in build-bud step of nodejs and nodejs loopback pipeline

tam512 commented 4 years ago

Use Kabanero v0.2.0
Tekton pipelines nodejs-build-deploy-pipeline and nodejs-loopback-build-deploy-pipeline run build-bud step take over 40 minutes to complete.

Example of Nodejs Loopback build-bud time:

name: build-bud
terminated:
containerID: 'cri-o://35d10d642b23d546b147471ee4b2a0591c2a70d7d4c40f62c47e311bf4654049'
exitCode: 0
finishedAt: '2019-11-06T15:35:46Z'
reason: Completed
startedAt: '2019-11-06T14:48:25Z'

tam512 commented 4 years ago

Attach the Nodejs Loopback build-bud log loopback-buildbud-try2.log.zip

marikaj123 commented 4 years ago

@tseelbach - Please help with assessing this issue and assign it to the appropriate owner.

tam512 commented 4 years ago

For nodejs-loopback-build-push-deploy-pipeline, the build step can not be completed, and the build run will fail with error PipelineRun "xxxxx-loopback-webhook-1573776386-2h6ds" failed to finish within "&Duration{Duration:1h0m0s,}"

kvijai82 commented 4 years ago

Looks like there is a fix in the latest buildah to improve performance of the copy step. Will pull that in and see if there is any improvement in this release.

groeges commented 4 years ago

There is a fix in the latest buildah that reduces the time taken to perform a COPY step within the docker file, however the appsody/appsody-buildah docker image that the pipelines uses is unable to utilise this later version as other fixes that are required (to prevent mount issues) have not been implemented. This means that we have to continue using buildah v1.9.0 until the fixes for the mount issue has been implemented in the later code and then a newer version of the appsody/appsody-buildah image will be generated for use by the pipelines.

marikaj123 commented 4 years ago

It is under debugging

marikaj123 commented 4 years ago

Gireesh and Kyle Christianson would need to help with this this issue.

kvijai82 commented 4 years ago

The node pipelines were failing with the latest buildah. Looks like there are some new fuse prereqs for it to work. @aadeshpa could you please look into this when you are back. Maybe first try to see if we get a performance improvement with the java ones and the new buildah. Thanks!

gireeshpunathil commented 4 years ago

is there any profile data available for me to inspect?
what is the baseline we are using to compare? when I was running against an OCP cluster it was a couple of minutes. what changed recently?
is there evidence that file copy is where the time is spent?

Per last discussion with Kyles, these were observed:

The buildah upgrade breaks appsody extract and bunch of other things
fix for that landed in fuse-overlay, not in buildah
fuse-overlay is an os component, not bundled in buildah
so a comprehensive solution is to upgrade buildah in the appsody-buildah image and fuse-overlay into the OS. The first one is easy to do, the second is not - as it involves changing every system that we provision / use.

I would suggest we:

define a baseline where the perf is good (in terms of few minutes)
figure out the step that takes long time (40 minutes) and the change that caused regression and take it from thereon

can you share the pipeline log that has timestamps, to start with?

aadeshpa commented 4 years ago

I tried testing pipelineruns(build-push-pipeline) using old appsody buildah image appsody/appsody-buildah:0.5.0-buildah1.9.0 and then with new image appsody/appsody-buildah:0.5.3 and found below results.

One good thing is with the new image appsody-buildah:0.5.3 we are able to run nodejs related pipelines where earlier we use to get error in Yum upgrade step.
However I did not see any performance improvement between the old image and new image. Below are the results.

Pipeline(build-push-pipeline)	Buildah image	Time
Java-mp	appsody/appsody-buildah:0.5.3	6 min 20
Java-mp	appsody/appsody-buildah:0.5.0-buildah1.9.0	6 min 22


nodejs-express	appsody/appsody-buildah:0.5.3	16 min
nodejs-express	appsody/appsody-buildah:0.5.0-buildah1.9.0	15 to 17 min


nodejs	appsody/appsody-buildah:0.5.3	12 min
nodejs	appsody/appsody-buildah:0.5.0-buildah1.9.0	12 min

cc: @gpunathi@in.ibm.com , @kvijai82

gireeshpunathil commented 4 years ago

thanks @aadeshpa . It would be interesting to see what are the values for loopback, for which this issue was raised originally.

aadeshpa commented 4 years ago

I tested with nodejs-loopback and below is the step in the logs where it stays stuck and then after 60 min, the pod itself terminates and the pipelinerun fails.

[Buildah] Installed:
[Buildah]   diffutils-3.6-5.el8.x86_64               libxkbcommon-0.8.2-1.el8.x86_64      
[Buildah]   xkeyboard-config-2.24-3.el8.noarch      
[Buildah] 
[Buildah] Complete!
[Buildah] 0 files removed
[Buildah] Finished installing dependencies
[Buildah] STEP 14: RUN groupadd --gid 1000 node   && useradd --uid 1000 --gid node --shell /bin/bash --create-home node
[Buildah] STEP 15: RUN mkdir ~/.npm-global
[Buildah] STEP 16: ENV NPM_CONFIG_PREFIX=~/.npm-global
[Buildah] STEP 17: COPY --chown=node:node . /project

then at step

[Buildah] STEP 18: COPY --chown=node:node --from=0 /project/node_modules /project/node_modules

nodejs-loopback	appsody/appsody-buildah:0.5.3	timeout 60 min(pipelinerun failed)
nodejs-loopback	appsody/appsody-buildah:0.5.0-buildah1.9.0	timeout 60 min (pipelinerun failed)

gireeshpunathil commented 4 years ago

thanks @aadeshpa . I am able to recreate this outside of the pipeline, and have raised a detailed bug report with buildah community.

the problem seems to raise from the fact that we have huge (~200 MB) node_modules folder belonging to the `loopback dependency to be copied to the image.

As we are building the dependency inside the Dockerfile , I am not sure why this folder should be there in the first place - looks like it is part of the original template? @raymondfeng

schiwitz commented 4 years ago

I ran into this issue while building a 1G container. I used buildah --debug bud . . to verify and saw the .local files where getting put in the container so the buildah process would. run until the filesystem filled. I put the .local in .dockerignore and was able to buildah bud the image.

gireeshpunathil commented 4 years ago

@schiwitz - thanks for the info. I tried your workaround, but does not look like it resolving in this case.

[pid 15587] 1577285784.432003 write(2, "time=\"2019-12-25T06:56:24-08:00\""..., 253time="2019-12-25T06:56:24-08:00" level=debug msg="copyFileWithTar(/root/.appsody/extract/lp/temp/foo/1/0/2/foo.txt, /var/lib/containers/storage/overlay/71ff4c26e61de5d4485815e9c2fad83938899979bbe2e7a5ff80b8fdd79e5469/merged/project/foo/1/0/2/foo.txt)"

...

[pid 15587] 1577285784.570250 write(2, "time=\"2019-12-25T06:56:24-08:00\""..., 253time="2019-12-25T06:56:24-08:00" level=debug msg="copyFileWithTar(/root/.appsody/extract/lp/temp/foo/1/0/3/foo.txt, /var/lib/containers/storage/overlay/71ff4c26e61de5d4485815e9c2fad83938899979bbe2e7a5ff80b8fdd79e5469/merged/project/foo/1/0/3/foo.txt)"

to process one file, looks like it takes ~150 ms - which seems to be too much, and when that gets multiplied we get into the issue. The loopback project has ~17K nested files / folders!

aadeshpa commented 4 years ago

@gireeshpunathil : I am trying to increase the timeout from 60 min to 120 in the pipelinerun file just for testing, and will let you know what is the exact time it takes to run the nodejs-loopback pipeline successfully.

gireeshpunathil commented 4 years ago

thanks @aadeshpa . While that number would be of some use, anything that takes more than 5 minutes in this use case is not usable in production, according to me - as we are not talking about GBs, and we are also not talking about network transfer, instead 200 MB from disk-to-memory-to-disk.

aadeshpa commented 4 years ago

@gireeshpunathil : After increasing the timeout, below is the result

nodejs-loopback	appsody/appsody-buildah:0.5.3	85 min with completion.
nodejs-loopback	appsody/appsody-buildah:0.5.0-buildah1.9.0	86 min with completion

gireeshpunathil commented 4 years ago

ok, thanks for the news. So, it is indeed moving!

aadeshpa commented 4 years ago

This is held up on the upstream issue with buildah, and the issue is linked above by @gireeshpunathil which is #2047 under buildah repository.

marikaj123 commented 4 years ago

According to Gireesh Punathil: The issue was isolated and raised it with buildah project: https://github.com/containers/buildah/issues/2047 Gireesh Punathil: a potential fix being worked out in https://github.com/containers/buildah/pull/2070

marikaj123 commented 4 years ago

According to Gireesh: we are good in terms of the debugging, and root cause analysis. The fix is in the master (of buildah). They promised to cut a release early next week, which we can consume. On the other hand, we may have a challenge in terms of upgrading buildah. I am still making an assessment on that front, and will come up with my observations, before evening

gireeshpunathil commented 4 years ago

the issue being resolved, verified (by me), a fix devised and a release is planned for next week, through https://quay.io/repository/buildah/stable?tab=tags. that is going to resolve the reported performance issue for loopback pipeline sequences.

but this brings a new challenge. Those who are familiar in this area may recollect the issue reported https://github.com/containers/fuse-overlayfs/issues/116 that inhibited bind mounts with certain types of file systems in the host, and was fixed in fuse-overlayfs v0.7. reference: https://github.com/containers/fuse-overlayfs/issues/116#issuecomment-556310655

I did some study around this, and my current understanding is that the host needs to install / upgrade fuse-overlayfs, not the buildah container (container is already equipped with the right fuse-overlayfs binary, alongside the buildah).

According to https://github.com/containers/fuse-overlayfs/issues/116#issuecomment-556995963 , fuse-overlayfs v0.7 is available in fedora 30/31 , but it is possible that most of the build / test systems are older (the ones that I typically use are RHEL 7.6 / fedora 18)

So this means for this fix to work out of the box, we either need to use latest fedora systems, or / and document this.

I will update here when the release is available, request those who have test systems to test and update how it goes. @aadeshpa @tam512 @kvijai82 or anyone else!

/cc @marikaj123

marikaj123 commented 4 years ago

According to Tam Dinh Gireesh Punathil I have a new OCP 4.2 cluster with Kabanaro 0.5.0-rc2 installed. It uses image: ‘appsody/appsody-buildah:0.5.3’. The nodejs pipeline run passes, but loopback still fails with PipelineRunTimeout For nodejs-build-push-deploy-pipeline , the build step took about 9 minutes Container status: container: step-build imageID: >- docker.io/appsody/appsody-buildah@sha256:5553dcab0de2ee2be863694c64245f95124231e7cff4711aec41354074b78706 name: build terminated: containerID: 'cri-o://730bccb706f20b892939d07063ee72f8699de8166915c95b1bc949a2e9939bf1' exitCode: 0 finishedAt: '2020-01-15T18:32:02Z' reason: Completed startedAt: '2020-01-15T18:23:23Z' For nodejs-loopback-build-push-deploy-pipeline, it still fails with PipelineRun “nodejs-loopback-build-push-deploy-pipeline-run-s55ts” failed to finish within “&Duration{Duration:1h0m0s,}” Container status: container: step-build imageID: >- docker.io/appsody/appsody-buildah@sha256:5553dcab0de2ee2be863694c64245f95124231e7cff4711aec41354074b78706 name: build terminated: containerID: 'cri-o://cbc3753ad2da2707d5b964b94a93667b8682ecca58cad427839148d1237c974c' exitCode: 0 finishedAt: '2020-01-15T17:09:36Z' reason: Completed startedAt: '2020-01-15T16:11:08Z' Even though it took a long time to build loopback, the PipelineRuns status shows as failed, but everything shows as green inside the build-push-deploy-task step --------------------------------------------------

marikaj123 commented 4 years ago

Response from Gireesh: Tam - the fix https://github.com/containers/buildah/issues/2047 for the perf issue is not yet released upstream (buildah community). we are waiting for that. I will update this forum when that is available. w.r.t the status showing green, pls raise it with the team who is looking at the rendering, I am not part of that work.

gireeshpunathil commented 4 years ago

a buildah image has been published yesterday, that has the fix for this issue.
I have tested with my local recreate, and it works.
I have pushed a test container gireeshpunathil/appsody-buildah:v1 with that

@aadeshpa @tam512 or any one else: can you pls test this with the problematic pipelines? specifically for nodejs-build-deploy-pipeline and nodejs-loopback-build-deploy-pipeline for performance, and nodejs-build-push-deploy-pipeline for regression / consistency?

important caveats:

the buildah upgrade brings up the reported bind mount issue on certain systems. fedora 31 (> RHEL 8 I guess) carries the fix in the OS.
for older systems, we might need to upgrade fuse-overlay in the host.

We would need to figure out where it works and where it does not, and document this behavior.

marikaj123 commented 4 years ago

@aadeshpa and the pipeline team is working on testing the patch before Gereeish would merge the fix.

aadeshpa commented 4 years ago

we are still in process of verifying this issue, we will provide update here once we are done testing.

aadeshpa commented 4 years ago

@gireeshpunathil I am testing the pipelines in 0.4.0 collections with the image appsody/appsody-buildah:0.5.4 (which was created 13 days ago) since last one we used in our pipelines was appsody/appsody-buildah:0.5.3,please confirm if that is the image we should be using to test it based on this link of the images : https://hub.docker.com/r/appsody/appsody-buildah/tags

It feels as per your above comment that latest image you were saying was 4 days ago with the fix which seems to be buildah stable image from quay.io, however i see appsody/appsody-buildah:0.5.4 is 13 days ago, so is it that appsody-buildah would be updating their image with the fix yet. I am not clear on this but going ahead with the testing.

below are the results with appsody/appsody-buildah:0.5.4:

Pipeline(build-push-pipeline)	Buildah image	Time
nodejs-express	appsody/appsody-buildah:0.5.3	16 min
--	--	--
nodejs-express	appsody/appsody-buildah:0.5.4	13 to 14 min
--	--	--
nodejs-loopback	appsody/appsody-buildah:0.5.3	85 min
--	--	--
nodejs-loopback	appsody/appsody-buildah:0.5.4	55 to 57 min

There are some changes in the timings, but it does not look completely solved with the appsody/appsody-buildah:0.5.4 image

aadeshpa commented 4 years ago

After discussion with @kvijai82 I understood that we would need to test this with the test container given by @gireeshpunathil gireeshpunathil/appsody-buildah:v1. Will test and report the result for the pipelinerun run time.

aadeshpa commented 4 years ago

@gireeshpunathil : I tested with the test container (gireeshpunathil/appsody-buildah:v1) you gave as per above comments and ran nodejs-express pipelinerun and we are getting the old Yum upgrade error , @gireeshpunathil is this the fuse overlay issue you were saying could be the side effect of the change ?

com --label org.opencontainers.image.title=git-source --label org.opencontainers.image.created=2020-01-21T21:55:46Z -f /builder/home/.appsody/extract/git-source/Dockerfile /builder/home/.appsody/extract/git-source
[Buildah] STEP 1: FROM registry.access.redhat.com/ubi8/nodejs-10:1-54
[Buildah] Getting image source signatures
[Buildah] Copying blob sha256:dc0478cb20c080aa5d428457aee1ed802fdf477589304bb5791bf6aa40c58aa5
[Buildah] Copying blob sha256:0e8ea260d0262eac3725175d3d499ead6fd77cb1fa8272b3e665e8f64044fb89
[Buildah] Copying blob sha256:2e37d1e56071c6902b26ae4728b962da1090983c743e3a17b5dd589d73a8dcef
[Buildah] Copying blob sha256:340ff6d7f58c908c438ce89bb845caee3649de828a81569317ed0fe169a97de2
[Buildah] Copying blob sha256:9c533c1f5da8cdca73c195234acf62c6fbe59afcd378b39bdd3929fce8c6f9f0
[Buildah] Copying config sha256:1bf9700289a45148a1a4b3b5a4f6fe458bceb374b53e475e545d548414666ef3
[Buildah] Writing manifest to image destination
[Buildah] Storing signatures
[Buildah] STEP 2: USER root
[Buildah] STEP 3: RUN yum upgrade --disableplugin=subscription-manager -y  && yum clean --disableplugin=subscription-manager packages  && echo 'Finished installing dependencies'
[Buildah] Red Hat Universal Base Image 8 (RPMs) - BaseOS  374 kB/s | 760 kB     00:02    
[Buildah] Red Hat Universal Base Image 8 (RPMs) - AppStre 1.4 MB/s | 3.1 MB     00:02    
[Buildah] Red Hat Universal Base Image 8 (RPMs) - CodeRea 7.2 kB/s | 9.1 kB     00:01    
[Buildah] Dependencies resolved.
[Buildah] ================================================================================
[Buildah]  Package             Arch      Version                 Repository          Size
[Buildah] ================================================================================
[Buildah] Upgrading:
[Buildah]  systemd-libs        x86_64    239-18.el8_1.1          ubi-8-baseos       560 k
[Buildah]  setup               noarch    2.12.2-2.el8_1.1        ubi-8-baseos       180 k
[Buildah]  systemd             x86_64    239-18.el8_1.1          ubi-8-baseos       3.5 M
[Buildah]  ca-certificates     noarch    2019.2.32-80.0.el8_1    ubi-8-baseos       407 k
[Buildah]  kernel-headers      x86_64    4.18.0-147.3.1.el8_1    ubi-8-baseos       2.7 M
[Buildah]  systemd-pam         x86_64    239-18.el8_1.1          ubi-8-baseos       230 k
[Buildah]  gdb-gdbserver       x86_64    8.2-6.el8_0             ubi-8-appstream    435 k
[Buildah]  wget                x86_64    1.19.5-8.el8_1.1        ubi-8-appstream    735 k
[Buildah]  git-core            x86_64    2.18.2-1.el8_1          ubi-8-appstream    5.0 M
[Buildah]  git-core-doc        noarch    2.18.2-1.el8_1          ubi-8-appstream    2.3 M
[Buildah]  perl-Git            noarch    2.18.2-1.el8_1          ubi-8-appstream     77 k
[Buildah]  git                 x86_64    2.18.2-1.el8_1          ubi-8-appstream    186 k
[Buildah] Installing dependencies:
[Buildah]  xkeyboard-config    noarch    2.24-3.el8              ubi-8-appstream    828 k
[Buildah] Installing weak dependencies:
[Buildah]  diffutils           x86_64    3.6-5.el8               ubi-8-baseos       359 k
[Buildah]  libxkbcommon        x86_64    0.8.2-1.el8             ubi-8-appstream    116 k
[Buildah] 
[Buildah] Transaction Summary
[Buildah] ================================================================================
[Buildah] Install   3 Packages
[Buildah] Upgrade  12 Packages
[Buildah] 
[Buildah] Total download size: 17 M
[Buildah] Downloading Packages:
[Buildah] (1/15): libxkbcommon-0.8.2-1.el8.x86_64.rpm     108 kB/s | 116 kB     00:01    
[Buildah] (2/15): diffutils-3.6-5.el8.x86_64.rpm          277 kB/s | 359 kB     00:01    
[Buildah] (3/15): xkeyboard-config-2.24-3.el8.noarch.rpm  581 kB/s | 828 kB     00:01    
[Buildah] (4/15): setup-2.12.2-2.el8_1.1.noarch.rpm       718 kB/s | 180 kB     00:00    
[Buildah] (5/15): systemd-libs-239-18.el8_1.1.x86_64.rpm  901 kB/s | 560 kB     00:00    
[Buildah] (6/15): ca-certificates-2019.2.32-80.0.el8_1.no 998 kB/s | 407 kB     00:00    
[Buildah] (7/15): systemd-pam-239-18.el8_1.1.x86_64.rpm   962 kB/s | 230 kB     00:00    
[Buildah] (8/15): systemd-239-18.el8_1.1.x86_64.rpm       3.7 MB/s | 3.5 MB     00:00    
[Buildah] (9/15): gdb-gdbserver-8.2-6.el8_0.x86_64.rpm    1.1 MB/s | 435 kB     00:00    
[Buildah] (10/15): kernel-headers-4.18.0-147.3.1.el8_1.x8 3.0 MB/s | 2.7 MB     00:00    
[Buildah] (11/15): wget-1.19.5-8.el8_1.1.x86_64.rpm       1.9 MB/s | 735 kB     00:00    
[Buildah] (12/15): perl-Git-2.18.2-1.el8_1.noarch.rpm     444 kB/s |  77 kB     00:00    
[Buildah] (13/15): git-2.18.2-1.el8_1.x86_64.rpm          1.0 MB/s | 186 kB     00:00    
[Buildah] (14/15): git-core-doc-2.18.2-1.el8_1.noarch.rpm 3.6 MB/s | 2.3 MB     00:00    
[Buildah] (15/15): git-core-2.18.2-1.el8_1.x86_64.rpm     4.4 MB/s | 5.0 MB     00:01    
[Buildah] --------------------------------------------------------------------------------
[Buildah] Total                                           4.7 MB/s |  17 MB     00:03     
[Buildah] Running transaction check
[Buildah] Transaction check succeeded.
[Buildah] Running transaction test
[Buildah] Transaction test succeeded.
[Buildah] Running transaction
[Buildah]   Preparing        :                                                        1/1 
[Buildah]   Running scriptlet: git-core-2.18.2-1.el8_1.x86_64                         1/1 
[Buildah]   Upgrading        : git-core-2.18.2-1.el8_1.x86_64                        1/27Error unpacking rpm package git-core-2.18.2-1.el8_1.x86_64
[Buildah]  
[Buildah]   Upgrading        : git-core-doc-2.18.2-1.el8_1.noarch                    2/27 
[Buildah] error: unpacking of archive failed on file /usr/bin/git-receive-pack;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: git-core-2.18.2-1.el8_1.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package git-core-doc-2.18.2-1.el8_1.noarch
[Buildah]   Upgrading        : perl-Git-2.18.2-1.el8_1.noarch                        3/27 
[Buildah] error: unpacking of archive failed on file /usr/share/doc/git/contrib/hooks;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: git-core-doc-2.18.2-1.el8_1.noarch: install failed
[Buildah] 
[Buildah]   Upgrading        : git-2.18.2-1.el8_1.x86_64                             4/27Error unpacking rpm package git-2.18.2-1.el8_1.x86_64
[Buildah]  
[Buildah]   Upgrading        : systemd-libs-239-18.el8_1.1.x86_64                    5/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/02/6f3da3634aad3755690d3c851a495241a5c51f;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: git-2.18.2-1.el8_1.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package systemd-libs-239-18.el8_1.1.x86_64
[Buildah]   Installing       : xkeyboard-config-2.24-3.el8.noarch                    6/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/35/51d9b032e7f9ba2bfc35ccd1fc45949edb8890;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: systemd-libs-239-18.el8_1.1.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package xkeyboard-config-2.24-3.el8.noarch
[Buildah]   Installing       : libxkbcommon-0.8.2-1.el8.x86_64                       7/27 
[Buildah] error: unpacking of archive failed on file /usr/share/X11/xkb/rules/xorg;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: xkeyboard-config-2.24-3.el8.noarch: install failed
[Buildah] 
[Buildah] Error unpacking rpm package libxkbcommon-0.8.2-1.el8.x86_64
[Buildah]   Installing       : diffutils-3.6-5.el8.x86_64                            8/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/58/3d03f1b1ccadb43a0d2e462bd8159da1907f83;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: libxkbcommon-0.8.2-1.el8.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package diffutils-3.6-5.el8.x86_64
[Buildah]   Upgrading        : systemd-pam-239-18.el8_1.1.x86_64                     9/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/06/943219e8d31d7b531f192331a12d28a3ff1b14;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: diffutils-3.6-5.el8.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package systemd-pam-239-18.el8_1.1.x86_64
[Buildah]   Running scriptlet: systemd-239-18.el8_1.1.x86_64                        10/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/52/af509bece6ee8a2d31d44c068c67c0e4f1f78c;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: systemd-pam-239-18.el8_1.1.x86_64: install failed
[Buildah] 
[Buildah]   Upgrading        : systemd-239-18.el8_1.1.x86_64                        10/27Error unpacking rpm package systemd-239-18.el8_1.1.x86_64
[Buildah]  
[Buildah]   Upgrading        : wget-1.19.5-8.el8_1.1.x86_64                         11/27 
[Buildah] error: unpacking of archive failed on file /etc/rc.local;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: systemd-239-18.el8_1.1.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package wget-1.19.5-8.el8_1.1.x86_64
[Buildah]   Upgrading        : gdb-gdbserver-8.2-6.el8_0.x86_64                     12/27 
[Buildah] error: unpacking of archive failed on file /usr/lib/.build-id/18/333ad81a17b7899aa7e3cd6758b2f81aaeca82;5e277376: cpio: utime failed - Inappropriate ioctl for device
[Buildah] error: wget-1.19.5-8.el8_1.1.x86_64: install failed
[Buildah] 
[Buildah] Error unpacking rpm package gdb-gdbserver-8.2-6.el8_0.x86_64

aadeshpa commented 4 years ago

We verified for Java-mp pipelinerun with the test container image from @gireeshpunathil and found below result with it

Pipeline(build-push-pipeline)	Buildah image	Time
java-mp	appsody/appsody-buildah:0.5.3	6 min 20 sec(earlier result)
--	--	--
java-mp	gireeshpunathil/appsody-buildah:v1	5min 40 sec

gireeshpunathil commented 4 years ago

@aadeshpa - thanks for the tests.

is this the fuse overlay issue you were saying

can't say for sure. can you please:

share the host details? cat /etc/os-release
share the steps to reproduce?

aadeshpa commented 4 years ago

@gireeshpunathil : below are details of the fyre env we use.

[root@foxing-inf ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

aadeshpa commented 4 years ago

After we discussed with @gireeshpunathil he confirmed that we have confirmation from the fuse-overlayfs community that RHEL 7 is vulnerable for the reported issue. the fix is only made available in fedora 31 / or RHEL compatible versions ( which is 8 and above)

I will be trying to upgrade the fyre env from 7.6 RHEL to 8.0 with the help of this article I found:- https://www.tecmint.com/upgrade-from-rhel-7-to-rhel-8/

If anyone has idea of how to upgrade to 8.0 the fyre env we get, let us know.

tseelbach commented 4 years ago

FYI this issue has insights into the buildah / fuse-overlayfs problem. https://github.com/containers/buildah/issues/1849

it also say that updating to 7.8 will help https://github.com/containers/buildah/issues/1849#issuecomment-551077682

gireeshpunathil commented 4 years ago

ok, given

the impact of this issue,
and a bit of confusion caused due to many issues reported upstream that may or may not be related leading to unertainty about which package / OS versions contain the fix,

I suggest the following:

we test our use case on a wide variety of platforms [ rhel / centos / ubuntu / any other?] where fuse bug apply
for ease of testing, use the simple recreate, listed in the first comment of https://github.com/containers/buildah/issues/2047
figure out which one works out of the box, and which one works with package (fuse-verlay) upgrade
confirm correctness with full-blown pipeline testing in representative platforms, with lowest working versions
document the platform requirement, in https://github.com/kabanero-io/kabanero-pipelines 's readme

if people agree, here is a matrix. Feel free to edit or add your entries if you have tested on that combination.

OS	with vanilla os	with fuse upgrade	person tested
centos 7.6	BAD	BAD	gireesh
centos 8.0	GOOD	NA	gireesh
rhel 7.6	BAD	BAD	gireesh
rhel 7.7	BAD	BAD	gireesh
rhel 7.8
rhel 8.0	GOOD	NA	gireesh
rhel 8.1	GOOD	NA	gireesh
ubnt 16.04	BAD	BAD	gireesh
ubnt 18.04	GOOD	NA	gireesh

In general, I found only Linux kernel v4.18.0.x and above seem to be supporting this, with an exception of Ubuntu, where it works with v4.15.0-66 too. This is roughly inline with fuse-overlayfs requirements

aadeshpa commented 4 years ago

@gireeshpunathil : I tried following the steps in the article of the upgrade of fyre env from RHEL 7.6 to 8 and got stuck at this point .

[root@suffuse-inf manual-pipeline-runs]# subscription-manager attach --auto
This system is not yet registered. Try 'subscription-manager register --help' for more information.

aadeshpa commented 4 years ago

This link tells how to register the Red Hat subscription but do not have any login to Red Hat Customer Portal

https://www.tecmint.com/enable-redhat-subscription-reposiories-and-updates-for-rhel-7/

aadeshpa commented 4 years ago

I was able to move ahead with the below steps but I am stuck again.

Steps followed:

Goto https://ftp3.linux.ibm.com/register.php and request access to login that server.
once you have those credentials you should follow directions as described here: https://ftp3.linux.ibm.com/rhsm.php.
Start from the step number 3 from this link in the RHEL 7.6 on Fyre : https://www.tecmint.com/upgrade-from-rhel-7-to-rhel-8/

I was successful till step 7 and then I got stuck in step 8 in above link as the wget downloads the file however it cannot untar it.

[root@suffuse-inf ~]# wget https://access.redhat.com/sites/default/files/attachments/leapp-data3.tar.gz
--2020-01-27 13:39:54--  https://access.redhat.com/sites/default/files/attachments/leapp-data3.tar.gz
Resolving access.redhat.com (access.redhat.com)... 104.91.43.26
Connecting to access.redhat.com (access.redhat.com)|104.91.43.26|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://access.redhat.com/articles/3664871 [following]
--2020-01-27 13:39:54--  https://access.redhat.com/articles/3664871
Reusing existing connection to access.redhat.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘leapp-data3.tar.gz’

    [ <=>                                                                                                                                                    ] 114,799      647KB/s   in 0.2s   

2020-01-27 13:39:54 (647 KB/s) - ‘leapp-data3.tar.gz’ saved [114799]

[root@suffuse-inf ~]# tar -xf leapp-data3.tar.gz 

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I figured out we could not download the leapp-data3.tar.gz from https://access.redhat.com/sites/default/files/attachments/leapp-data3.tar.gz since this requires a login of redhat with subscription. If we get that we can try to use the content from the zip file to proceed further.

Step 1 and Step 2 PDFs from this wiki explains how to get RedHat NFR subscriptions -- https://apps.na.collabserv.com/wikis/home?lang=en-us#!/wiki/W5ccbfeca27ac_4588_82ce_50757d2fd1c1/page/RedHat Request for approval is sent for my id. Will check once i get the approval whether i could download that above zip file.

gireeshpunathil commented 4 years ago

@aadeshpa - I am no OS upgrade specialist, but I recommend to drop that route and follow my recommendation in the previous comment, which is much easier to run and observe result. Once we have a set, you could setup cluster in those representative systems and test with minimal effort

jdmcclur commented 4 years ago

I tried pulling the gireeshpunathil/appsody-buildah:v1 image and hacking it into my OpenShift environment on CoreOS 4.2, but I am seeing the following problem when building node.js.

STEP 4: RUN groupadd --gid 1000 node   && useradd --uid 1000 --gid node --shell /bin/bash --create-home node
groupadd: /etc/group.239: lock file already used
groupadd: cannot lock /etc/group; try again later.
subprocess exited with status 10
subprocess exited with status 10
error building at STEP "RUN groupadd --gid 1000 node   && useradd --uid 1000 --gid node --shell /bin/bash --create-home node": exit status 10

aadeshpa commented 4 years ago

I went passed ahead of the error to download the leapp-data3.tar.gz and then run preupgrade step that gave below errors in the report so we could not proceed with the upgrade from RHEL 7.6 to 8.

============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt
A report has been generated at /var/log/leapp/leapp-preupgrade.log

============================================================
                       END OF REPORT                        
============================================================
[root@yoked-inf files]# cat /var/log/leapp/leapp-report.txt
Risk Factor: high (inhibitor)
Title: Unsupported network configuration
Summary: Detected multiple network interfaces using unstable kernel names (e.g. eth0, eth1). Upgrade process can not continue because stability of names can not be guaranteed. Please read the article at https://access.redhat.com/solutions/4067471 for more information.
----------------------------------------
Risk Factor: high
Title: Packages will not be installed
Summary: 4 packages will not be installed or upgraded due to repositories unknown to leapp:
- postfix-pcre
- postfix-sqlite
- libselinux-python3
- postfix-cdb
----------------------------------------
Risk Factor: low
Title: Grep has incompatible changes in the next major version
Summary: If a file contains data improperly encoded for the current locale, and this is discovered before any of the file's contents are output, grep now treats the file as binary.
The 'grep -P' no longer reports an error and exits when given invalid UTF-8 data. Instead, it considers the data to be non-matching.
In locales with multibyte character encodings other than UTF-8, grep -P now reports an error and exits instead of misbehaving.
When searching binary data, grep now may treat non-text bytes as line terminators. This can boost performance significantly.
The 'grep -z' no longer automatically treats the byte '\200' as binary data.
Context no longer excludes selected lines omitted because of -m. For example, 'grep "^" -m1 -A1' now outputs the first two input lines, not just the first line.

Remediation: [hint] Please update your scripts to be compatible with the changes.
----------------------------------------
Risk Factor: low
Title: OpenSSH configured with removed configuration Protocol
Summary: OpenSSH is configured with removed configuration option Protocol. If this used to be for enabling SSHv1, this is no longer supported in RHEL 8. Otherwise this option can be simply removed.
----------------------------------------
Risk Factor: high
Title: Packages not signed by Red Hat found in the system
Summary: The following packages have not been signed by Red Hat and may be removed in the upgrade process:
- falcon-sensor
- katello-ca-consumer-ltc-rhs72.raleigh.ibm.com
Remediation: [command] yum -y remove falcon-sensor katello-ca-consumer-ltc-rhs72.raleigh.ibm.com
----------------------------------------
Risk Factor: low
Title: ntpd and ntpdate configuration will be migrated
Summary: ntpd, ntpdate service(s) detected to be enabled and active
----------------------------------------
Risk Factor: high
Title: Difference in Python versions and support in RHEL 8
Summary: In RHEL 8, there is no 'python' command. Python 3 (backward incompatible) is the primary Python version and Python 2 is available with limited support and limited set of packages. Read more here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/#using-python3
Remediation: [hint] Please run "alternatives --set python /usr/bin/python3" after upgrade
----------------------------------------
Risk Factor: low
Title: Postfix has incompatible changes in the next major version
Summary: Postfix 3.x has so called "compatibility safety net" that runs Postfix programs with backwards-compatible default settings. It will log a warning whenever backwards-compatible default setting may be required for continuity of service. Based on this logging the system administrator can decide if any backwards-compatible settings need to be made permanent in main.cf or master.cf, before turning off the backwards-compatibility safety net.
The backward compatibility safety net is by default turned off in Red Hat Enterprise Linux 8.
It can be turned on by running:  "postconf -e compatibility_level=0
It can be turned off by running: "postconf -e compatibility_level=2

In the Postfix MySQL database client, the default "option_group" value has changed to "client", i.e. it now reads options from the [client] group from the MySQL configuration file. To disable it, set "option_group" to the empty string.

The postqueue command no longer forces all message arrival times to be reported in UTC. To get the old behavior, set TZ=UTC in main.cf:import_environment.

Postfix 3.2 enables elliptic curve negotiation. This changes the default smtpd_tls_eecdh_grade setting to "auto", and introduces a new parameter "tls_eecdh_auto_curves" with the names of curves that may be negotiated.

The "master.cf" chroot default value has changed from "y" (yes) to "n" (no). This applies to master.cf services where chroot field is not explicitly specified.

The "append_dot_mydomain" default value has changed from "yes" to "no". You may need changing it to "yes" if senders cannot use complete domain names in e-mail addresses.

The "relay_domains" default value has changed from "$mydestination" to the empty value. This could result in unexpected "Relay access denied" errors or ETRN errors, because now will postfix by default relay only for the localhost.

The "mynetworks_style" default value has changed from "subnet" to "host". This parameter is used to implement the "permit_mynetworks" feature. The change could result in unexpected "access denied" errors, because postfix will now by default trust only the local machine, not the remote SMTP clients on the same IP subnetwork.

----------------------------------------
Risk Factor: info
Title: SElinux disabled
Summary: SElinux disabled, continuing...
----------------------------------------
Risk Factor: low
Title: Some enabled RPM repositories are unknown to Leapp
Summary: The following repositories with Red Hat-signed packages are unknown to Leapp:
- Local-Extras
- Local-Base
And the following packages installed from those repositories may not be upgraded:
- keyutils
- openssh-clients
- nss-tools
- perl-libs
- dhcp-libs
- nss-pem
- efivar-libs
- libsmartcols
- rsyslog
- iwl105-firmware
- openssl-libs
- python-jinja2
- systemd
- selinux-policy
- libsolv
- gcc-gfortran
- perl-macros
- iwl135-firmware
- nspr
- libnfsidmap
- policycoreutils-python
- grub2
- iwl2000-firmware
- libverto-libevent
- gcc-c++
- policycoreutils
- python-IPy
- libquadmath
- freetype
- iwl6000g2a-firmware
- dbus-libs
- iwl7265-firmware
- dhclient
- python
- iwl6050-firmware
- systemd-libs
- iwl4965-firmware
- systemd-python
- iwl2030-firmware
- libgudev1
- gnutls
- util-linux
- NetworkManager-config-server
- python-markupsafe
- libquadmath-devel
- iproute
- libyaml
- dbus
- libpath_utils
- python-babel
- tuned
- krb5-libs
- net-tools
- teamd
- systemd-sysv
- kernel
- rpcbind
- scap-security-guide
- python2-libcomps
- libblkid
- grub2-common
- audit-libs
- aide
- nss-sysinit
- libgcc
- json-glib
- patch
- libmodulemd
- python-enum34
- rsync
- libref_array
- glibc
- device-mapper
- shadow-utils
- lvm2-libs
- libcgroup
- linux-firmware
- grub2-tools-extra
- python2-leapp
- ipset
- ntp
- iwl1000-firmware
- git
- leapp-repository-deps
- libbasicobjects
- wget
- kernel-tools
- glibc-common
- haproxy
- elfutils-libs
- subscription-manager-rhsm
- libjpeg-turbo
- libuuid
- openssl
- dhcp-common
- libgomp
- python2-libdnf
- iwl5000-firmware
- subscription-manager
- python-perf
- nss-softokn
- glib2
- mariadb-libs
- binutils
- bind-export-libs
- hesiod
- iwl100-firmware
- subscription-manager-rhsm-certificates
- tcp_wrappers
- dnf-data
- kernel-devel
- libstdc++
- libcollection
- sos
- python-libs
- grub2-tools
- iwl7260-firmware
- nss
- iwl5150-firmware
- vim-minimal
- libssh2
- dnf
- libgfortran
- libstdc++-devel
- ntpdate
- grub2-pc-modules
- perl
- tzdata
- NetworkManager-libnm
- microcode_ctl
- iwl3160-firmware
- NetworkManager
- bind-libs-lite
- procps-ng
- iwl6000-firmware
- setools-libs
- quota-nls
- openssh-server
- libdnf
- nfs-utils
- kexec-tools
- libevent
- libtirpc
- kernel-headers
- leapp-deps
- ipset-libs
- kernel-tools-libs
- device-mapper-event-libs
- ksh
- lvm2
- nss-util
- device-mapper-event
- autofs
- openssh
- selinux-policy-targeted
- cronie-anacron
- xfsprogs
- NetworkManager-team
- libcomps
- sudo
- grub2-pc
- unzip
- sysstat
- bind-license
- curl
- perl-Git
- compat-libtiff3
- libmount
- gcc
- leapp-repository
- libcurl
- audit-libs-python
- leapp-repository-sos-plugin
- librhsm
- iwl3945-firmware
- elfutils-libelf
- nss-softokn-freebl
- python2-dnf
- glibc-devel
- elfutils-default-yama-scope
- quota
- device-mapper-libs
- polkit
- perl-Pod-Escapes
- openldap
- augeas-libs
- lm_sensors-libs
- leapp
- librepo
- glibc-headers
- cronie
- checkpolicy
- libteam
- audit
- iwl6000g2b-firmware
- python2-hawkey
- NetworkManager-tui
- libsemanage-python
- gssproxy
- grub2-tools-minimal
- pciutils
- cpp
- autogen-libopts
- elfutils
- libini_config
Remediation: [hint] You can file a request to add this repository to the scope of in-place upgrades by filing a support ticket
----------------------------------------

gireeshpunathil commented 4 years ago

@aadeshpa - please see comments https://github.com/kabanero-io/kabanero-pipelines/issues/121#issuecomment-578009040 and https://github.com/kabanero-io/kabanero-pipelines/issues/121#issuecomment-579044448 . To resolve this issue quickly, we need to isolate platforms where it works and where it does not, out of the box or with tweaking, and based on count, the versions where it works etc. we may take a call about whether this approach (fuse bug fix and buildah bug fix) is acceptable or not.

Pipeline testing can be complex; and as you and @jdmcclur also have pointed out, there could be unrelated issues hiding somewhere else (OS upgrade issue / Dockerfile syntax issue etc). We want to isolate all of those and diagnose and resolve the performance + mount bug through this issue. For that, I recommend we follow https://github.com/kabanero-io/kabanero-pipelines/issues/121#issuecomment-578009040

marikaj123 commented 4 years ago

Looking for an env to test with. Requested a Fyre env to install Kabanero and try to use the Codeready containers on it. then try to install Kabanero to test.

aadeshpa commented 4 years ago

@marikaj123 We are in process to validate the fix for this which is in appsody/appsody-buildahimage however the point to be noted is if this fix is to be used we will surely updated pipeline with that new fixed image of appsody/appsody-buildah but there will be a manadatory requirement for user to update their host VM from RHEL 7.6 to RHEL 8.0 as per explained by @gireeshpunathil in earlier comments.

Below are the environment setup options we tried and first 3 were unsuccessful. Since we need RHEL 8.0 to test this fix from @gireeshpunathil and fyre env only provides the OCP4.2 cluster with RHEL 7.6:

We tried getting OCP4.2 cluster from fyre env, which gives RHEL 7.6 and to test this we need RHEL 8.0 so we tried upgrade path where we tried steps to upgrade inplace RHEL 7.6 to 8 and that was not successful.
Second thing we tried was requesting single node ember of RHEL 8.0 from fyre environment and tried running the test container with the fix from @gireeshpunathil and just tried running appsody build command to build the nodejs-express project but we are facing issues with this test.
Next thing we tried to do is to install kabanero in single node ember ofr RHEL 8.0 from fyre environment, but that install does not work on it.
(In progress)Next option to get the environment to test this is requesting single node FYRE ember with RHEL 8.0 and try to install the Code ready container which gives us OCP 4.2 installed in that container and then we would try to install kabanero in that container.

gireeshpunathil commented 4 years ago

I will debug item (2) of the above.

aadeshpa commented 4 years ago

4(continue from previous comment). we are working with @stevenschader , he is helping us setup an fyre env with RHEL 8.0 and code ready container which will give us OCP 4 in that container, and then we will try to install kabanero on top of it to test this fix.

aadeshpa commented 4 years ago

we were able to get fyre env with code ready container installed on top of RHEL 8.1 and on that CRC container which had OCP 4 in the container. I was successfully able to run our kabanero install in this environment. Now i ran first pipeline nodjes-exp on it and it was successful. @Steven Schader helped me get the fyre node setup with all that installed on it. cc: @smcclem @bsulliv I will use this environment to test the fix from @gireesh

aadeshpa commented 4 years ago

@gireeshpunathil : I tested with your test container gireeshpunathil/appsody-buildah:v1 by running the nodejs-express build-push pipeline in RHEL 8.1 and kabanero 0-5-0 collection and it still failed at Yum Upgrade step.

This means for nodejs-express pipeline we are getting same error result in RHEL 7.6 and RHEL 8.1 with Yum upgrade error with appsody buildah container having the fix.

Environment OS

[kevin@svtcrc-389745-1 manual-pipeline-runs]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)

Nodjes-express Pipelinerun log snipped with error

[Buildah] STEP 2: USER root
[Buildah] STEP 3: RUN yum upgrade --disableplugin=subscription-manager -y  && yum install --disableplugin=subscription-manager python2 -y  && yum clean --disableplugin=subscription-manager packages  && ln -s /usr/bin/python2 /usr/bin/python  && useradd --uid 1000 --gid 0 --shell /bin/bash --create-home node
[Buildah] Red Hat Universal Base Image 8 (RPMs) - BaseOS  1.2 MB/s | 760 kB     00:00    
[Buildah] Red Hat Universal Base Image 8 (RPMs) - AppStre 3.7 MB/s | 3.1 MB     00:00    
[Buildah] Red Hat Universal Base Image 8 (RPMs) - CodeRea  19 kB/s | 9.1 kB     00:00    
[Buildah] Dependencies resolved.
[Buildah] ================================================================================
[Buildah]  Package             Arch      Version                 Repository          Size
[Buildah] ================================================================================
[Buildah] Upgrading:
[Buildah]  systemd-libs        x86_64    239-18.el8_1.1          ubi-8-baseos       560 k
[Buildah]  setup               noarch    2.12.2-2.el8_1.1        ubi-8-baseos       180 k
[Buildah]  systemd             x86_64    239-18.el8_1.1          ubi-8-baseos       3.5 M
[Buildah]  libarchive          x86_64    3.3.2-8.el8_1           ubi-8-baseos       359 k
[Buildah]  ca-certificates     noarch    2019.2.32-80.0.el8_1    ubi-8-baseos       407 k
[Buildah]  sqlite              x86_64    3.26.0-4.el8_1          ubi-8-baseos       666 k
[Buildah]  sqlite-libs         x86_64    3.26.0-4.el8_1          ubi-8-baseos       579 k
[Buildah]  bsdtar              x86_64    3.3.2-8.el8_1           ubi-8-baseos        71 k
[Buildah]  sqlite-devel        x86_64    3.26.0-4.el8_1          ubi-8-baseos       163 k
[Buildah]  kernel-headers      x86_64    4.18.0-147.3.1.el8_1    ubi-8-baseos       2.7 M
[Buildah]  systemd-pam         x86_64    239-18.el8_1.1          ubi-8-baseos       230 k
[Buildah]  gdb-gdbserver       x86_64    8.2-6.el8_0             ubi-8-appstream    435 k
[Buildah]  wget                x86_64    1.19.5-8.el8_1.1        ubi-8-appstream    735 k
[Buildah]  git-core            x86_64    2.18.2-1.el8_1          ubi-8-appstream    5.0 M
[Buildah]  git-core-doc        noarch    2.18.2-1.el8_1          ubi-8-appstream    2.3 M
[Buildah]  perl-Git            noarch    2.18.2-1.el8_1          ubi-8-appstream     77 k
[Buildah]  git                 x86_64    2.18.2-1.el8_1          ubi-8-appstream    186 k
[Buildah] Installing dependencies:
[Buildah]  xkeyboard-config    noarch    2.24-3.el8              ubi-8-appstream    828 k
[Buildah] Installing weak dependencies:
[Buildah]  diffutils           x86_64    3.6-5.el8               ubi-8-baseos       359 k
[Buildah]  libxkbcommon        x86_64    0.8.2-1.el8             ubi-8-appstream    116 k
[Buildah] 
[Buildah] Transaction Summary
[Buildah] ================================================================================
[Buildah] Install   3 Packages
[Buildah] Upgrade  17 Packages
[Buildah] 
[Buildah] Total download size: 19 M
[Buildah] Downloading Packages:
[Buildah] (1/20): libxkbcommon-0.8.2-1.el8.x86_64.rpm     426 kB/s | 116 kB     00:00    
[Buildah] (2/20): diffutils-3.6-5.el8.x86_64.rpm          718 kB/s | 359 kB     00:00    
[Buildah] (3/20): setup-2.12.2-2.el8_1.1.noarch.rpm       1.0 MB/s | 180 kB     00:00    
[Buildah] (4/20): systemd-libs-239-18.el8_1.1.x86_64.rpm  1.3 MB/s | 560 kB     00:00    
[Buildah] (5/20): xkeyboard-config-2.24-3.el8.noarch.rpm  1.0 MB/s | 828 kB     00:00    
[Buildah] (6/20): libarchive-3.3.2-8.el8_1.x86_64.rpm     1.9 MB/s | 359 kB     00:00    
[Buildah] (7/20): ca-certificates-2019.2.32-80.0.el8_1.no 2.9 MB/s | 407 kB     00:00    
[Buildah] (8/20): sqlite-3.26.0-4.el8_1.x86_64.rpm        5.4 MB/s | 666 kB     00:00    
[Buildah] (9/20): sqlite-libs-3.26.0-4.el8_1.x86_64.rpm   4.9 MB/s | 579 kB     00:00    
[Buildah] (10/20): bsdtar-3.3.2-8.el8_1.x86_64.rpm        1.3 MB/s |  71 kB     00:00    
[Buildah] (11/20): sqlite-devel-3.26.0-4.el8_1.x86_64.rpm 2.7 MB/s | 163 kB     00:00    
[Buildah] (12/20): systemd-pam-239-18.el8_1.1.x86_64.rpm  2.9 MB/s | 230 kB     00:00    
[Buildah] (13/20): gdb-gdbserver-8.2-6.el8_0.x86_64.rpm   3.4 MB/s | 435 kB     00:00    
[Buildah] (14/20): kernel-headers-4.18.0-147.3.1.el8_1.x8 9.1 MB/s | 2.7 MB     00:00    
[Buildah] (15/20): wget-1.19.5-8.el8_1.1.x86_64.rpm       5.4 MB/s | 735 kB     00:00    
[Buildah] (16/20): systemd-239-18.el8_1.1.x86_64.rpm      3.4 MB/s | 3.5 MB     00:01    
[Buildah] (17/20): perl-Git-2.18.2-1.el8_1.noarch.rpm     1.3 MB/s |  77 kB     00:00    
[Buildah] (18/20): git-2.18.2-1.el8_1.x86_64.rpm          2.3 MB/s | 186 kB     00:00    
[Buildah] (19/20): git-core-doc-2.18.2-1.el8_1.noarch.rpm 3.7 MB/s | 2.3 MB     00:00    
[Buildah] (20/20): git-core-2.18.2-1.el8_1.x86_64.rpm     5.6 MB/s | 5.0 MB     00:00    
[Buildah] --------------------------------------------------------------------------------
[Buildah] Total                                           8.2 MB/s |  19 MB     00:02     
[Buildah] Running transaction check
[Buildah] Transaction check succeeded.
[Buildah] Running transaction test
[Buildah] Transaction test succeeded.
[Buildah] Running transaction
[Buildah]   Preparing        :                                                        1/1 
[Buildah]   Running scriptlet: git-core-2.18.2-1.el8_1.x86_64                         1/1 
[Buildah]   Upgrading        : git-core-2.18.2-1.el8_1.x86_64                        1/37 
[Buildah]   Upgrading        : sqlite-libs-3.26.0-4.el8_1.x86_64                     2/37 
[Buildah]   Upgrading        : sqlite-3.26.0-4.el8_1.x86_64                          3/37 
[Buildah]   Upgrading        : git-core-doc-2.18.2-1.el8_1.noarch                    4/37 
[Buildah]   Upgrading        : perl-Git-2.18.2-1.el8_1.noarch                        5/37 
[Buildah]   Upgrading        : git-2.18.2-1.el8_1.x86_64                             6/37 
[Buildah]   Upgrading        : libarchive-3.3.2-8.el8_1.x86_64                       7/37 
[Buildah]   Upgrading        : systemd-libs-239-18.el8_1.1.x86_64                    8/37 
[Buildah]   Running scriptlet: systemd-libs-239-18.el8_1.1.x86_64                    8/37 
[Buildah]   Installing       : xkeyboard-config-2.24-3.el8.noarch                    9/37 
[Buildah]   Installing       : libxkbcommon-0.8.2-1.el8.x86_64                      10/37 
[Buildah]   Installing       : diffutils-3.6-5.el8.x86_64                           11/37 
[Buildah]   Running scriptlet: diffutils-3.6-5.el8.x86_64                           11/37 
[Buildah]   Upgrading        : systemd-pam-239-18.el8_1.1.x86_64                    12/37 
[Buildah]   Running scriptlet: systemd-239-18.el8_1.1.x86_64                        13/37 
[Buildah]   Upgrading        : systemd-239-18.el8_1.1.x86_64                        13/37 
[Buildah]   Running scriptlet: systemd-239-18.el8_1.1.x86_64                        13/37 
[Buildah]   Upgrading        : bsdtar-3.3.2-8.el8_1.x86_64                          14/37 
[Buildah]   Upgrading        : sqlite-devel-3.26.0-4.el8_1.x86_64                   15/37 
[Buildah]   Upgrading        : wget-1.19.5-8.el8_1.1.x86_64                         16/37 
[Buildah]   Running scriptlet: wget-1.19.5-8.el8_1.1.x86_64                         16/37 
[Buildah]   Upgrading        : gdb-gdbserver-8.2-6.el8_0.x86_64                     17/37 
[Buildah]   Upgrading        : kernel-headers-4.18.0-147.3.1.el8_1.x86_64           18/37 
[Buildah]   Running scriptlet: ca-certificates-2019.2.32-80.0.el8_1.noarch          19/37 
[Buildah]   Upgrading        : ca-certificates-2019.2.32-80.0.el8_1.noarch          19/37 
[Buildah]   Running scriptlet: ca-certificates-2019.2.32-80.0.el8_1.noarch          19/37 
[Buildah]   Upgrading        : setup-2.12.2-2.el8_1.1.noarch                        20/37 
[Buildah] warning: /etc/shadow created as /etc/shadow.rpmnew
[Buildah] 
[Buildah]   Running scriptlet: setup-2.12.2-2.el8_1.1.noarch                        20/37 
[Buildah]   Cleanup          : sqlite-devel-3.26.0-3.el8.x86_64                     21/37 
[Buildah]   Cleanup          : perl-Git-2.18.1-4.el8.noarch                         22/37 
[Buildah]   Cleanup          : kernel-headers-4.18.0-147.el8.x86_64                 23/37 
[Buildah]   Cleanup          : ca-certificates-2018.2.24-6.el8.noarch               24/37 
[Buildah]   Cleanup          : setup-2.12.2-2.el8.noarch                            25/37 
[Buildah]   Cleanup          : git-2.18.1-4.el8.x86_64                              26/37 
[Buildah]   Cleanup          : bsdtar-3.3.2-7.el8.x86_64                            27/37 
[Buildah]   Running scriptlet: systemd-239-18.el8.x86_64                            28/37 
[Buildah]   Cleanup          : systemd-239-18.el8.x86_64                            28/37 
[Buildah]   Cleanup          : git-core-doc-2.18.1-4.el8.noarch                     29/37 
[Buildah]   Cleanup          : sqlite-3.26.0-3.el8.x86_64                           30/37 
[Buildah]   Cleanup          : sqlite-libs-3.26.0-3.el8.x86_64                      31/37 
[Buildah]   Cleanup          : git-core-2.18.1-4.el8.x86_64                         32/37 
[Buildah]   Cleanup          : systemd-libs-239-18.el8.x86_64                       33/37 
[Buildah]   Cleanup          : systemd-pam-239-18.el8.x86_64                        34/37 
[Buildah]   Cleanup          : libarchive-3.3.2-7.el8.x86_64                        35/37 
[Buildah]   Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64                         36/37 
[Buildah]   Cleanup          : wget-1.19.5-7.el8_0.1.x86_64                         36/37 
[Buildah]   Cleanup          : gdb-gdbserver-8.2-6.el8.x86_64                       37/37 
[Buildah]   Running scriptlet: gdb-gdbserver-8.2-6.el8.x86_64                       37/37 
[Buildah]   Running scriptlet: systemd-239-18.el8_1.1.x86_64                        37/37 
[Buildah]   Verifying        : diffutils-3.6-5.el8.x86_64                            1/37 
[Buildah]   Verifying        : libxkbcommon-0.8.2-1.el8.x86_64                       2/37 
[Buildah]   Verifying        : xkeyboard-config-2.24-3.el8.noarch                    3/37 
[Buildah]   Verifying        : systemd-libs-239-18.el8_1.1.x86_64                    4/37 
[Buildah]   Verifying        : systemd-libs-239-18.el8.x86_64                        5/37 
[Buildah]   Verifying        : setup-2.12.2-2.el8_1.1.noarch                         6/37 
[Buildah]   Verifying        : setup-2.12.2-2.el8.noarch                             7/37 
[Buildah]   Verifying        : systemd-239-18.el8_1.1.x86_64                         8/37 
[Buildah]   Verifying        : systemd-239-18.el8.x86_64                             9/37 
[Buildah]   Verifying        : libarchive-3.3.2-8.el8_1.x86_64                      10/37 
[Buildah]   Verifying        : libarchive-3.3.2-7.el8.x86_64                        11/37 
[Buildah]   Verifying        : ca-certificates-2019.2.32-80.0.el8_1.noarch          12/37 
[Buildah]   Verifying        : ca-certificates-2018.2.24-6.el8.noarch               13/37 
[Buildah]   Verifying        : sqlite-3.26.0-4.el8_1.x86_64                         14/37 
[Buildah]   Verifying        : sqlite-3.26.0-3.el8.x86_64                           15/37 
[Buildah]   Verifying        : sqlite-libs-3.26.0-4.el8_1.x86_64                    16/37 
[Buildah]   Verifying        : sqlite-libs-3.26.0-3.el8.x86_64                      17/37 
[Buildah]   Verifying        : bsdtar-3.3.2-8.el8_1.x86_64                          18/37 
[Buildah]   Verifying        : bsdtar-3.3.2-7.el8.x86_64                            19/37 
[Buildah]   Verifying        : sqlite-devel-3.26.0-4.el8_1.x86_64                   20/37 
[Buildah]   Verifying        : sqlite-devel-3.26.0-3.el8.x86_64                     21/37 
[Buildah]   Verifying        : kernel-headers-4.18.0-147.3.1.el8_1.x86_64           22/37 
[Buildah]   Verifying        : kernel-headers-4.18.0-147.el8.x86_64                 23/37 
[Buildah]   Verifying        : systemd-pam-239-18.el8_1.1.x86_64                    24/37 
[Buildah]   Verifying        : systemd-pam-239-18.el8.x86_64                        25/37 
[Buildah]   Verifying        : gdb-gdbserver-8.2-6.el8_0.x86_64                     26/37 
[Buildah]   Verifying        : gdb-gdbserver-8.2-6.el8.x86_64                       27/37 
[Buildah]   Verifying        : wget-1.19.5-8.el8_1.1.x86_64                         28/37 
[Buildah]   Verifying        : wget-1.19.5-7.el8_0.1.x86_64                         29/37 
[Buildah]   Verifying        : git-core-2.18.2-1.el8_1.x86_64                       30/37 
[Buildah]   Verifying        : git-core-2.18.1-4.el8.x86_64                         31/37 
[Buildah]   Verifying        : git-core-doc-2.18.2-1.el8_1.noarch                   32/37 
[Buildah]   Verifying        : git-core-doc-2.18.1-4.el8.noarch                     33/37 
[Buildah]   Verifying        : perl-Git-2.18.2-1.el8_1.noarch                       34/37 
[Buildah]   Verifying        : perl-Git-2.18.1-4.el8.noarch                         35/37 
[Buildah]   Verifying        : git-2.18.2-1.el8_1.x86_64                            36/37 
[Buildah]   Verifying        : git-2.18.1-4.el8.x86_64                              37/37 
[Buildah] Installed products updated.
[Buildah] 
[Buildah] Upgraded:
[Buildah]   systemd-libs-239-18.el8_1.1.x86_64                                            
[Buildah]   setup-2.12.2-2.el8_1.1.noarch                                                 
[Buildah]   systemd-239-18.el8_1.1.x86_64                                                 
[Buildah]   libarchive-3.3.2-8.el8_1.x86_64                                               
[Buildah]   ca-certificates-2019.2.32-80.0.el8_1.noarch                                   
[Buildah]   sqlite-3.26.0-4.el8_1.x86_64                                                  
[Buildah]   sqlite-libs-3.26.0-4.el8_1.x86_64                                             
[Buildah]   bsdtar-3.3.2-8.el8_1.x86_64                                                   
[Buildah]   sqlite-devel-3.26.0-4.el8_1.x86_64                                            
[Buildah]   kernel-headers-4.18.0-147.3.1.el8_1.x86_64                                    
[Buildah]   systemd-pam-239-18.el8_1.1.x86_64                                             
[Buildah]   gdb-gdbserver-8.2-6.el8_0.x86_64                                              
[Buildah]   wget-1.19.5-8.el8_1.1.x86_64                                                  
[Buildah]   git-core-2.18.2-1.el8_1.x86_64                                                
[Buildah]   git-core-doc-2.18.2-1.el8_1.noarch                                            
[Buildah]   perl-Git-2.18.2-1.el8_1.noarch                                                
[Buildah]   git-2.18.2-1.el8_1.x86_64                                                     
[Buildah] 
[Buildah] Installed:
[Buildah]   diffutils-3.6-5.el8.x86_64               libxkbcommon-0.8.2-1.el8.x86_64      
[Buildah]   xkeyboard-config-2.24-3.el8.noarch      
[Buildah] 
[Buildah] Complete!
[Buildah] Last metadata expiration check: 0:01:03 ago on Thu Jan 30 19:53:46 2020.
[Buildah] Dependencies resolved.
[Buildah] ================================================================================
[Buildah]  Package    Arch   Version                                Repository       Size
[Buildah] ================================================================================
[Buildah] Installing:
[Buildah]  python2    x86_64 2.7.16-12.module+el8.1.0+4148+33a50073 ubi-8-appstream 109 k
[Buildah] Installing dependencies:
[Buildah]  python2-libs
[Buildah]             x86_64 2.7.16-12.module+el8.1.0+4148+33a50073 ubi-8-appstream 6.0 M
[Buildah]  python2-setuptools-wheel
[Buildah]             noarch 39.0.1-11.module+el8.1.0+3446+c3d52da3 ubi-8-appstream 289 k
[Buildah]  python2-pip-wheel
[Buildah]             noarch 9.0.3-14.module+el8.1.0+3446+c3d52da3  ubi-8-appstream 1.2 M
[Buildah] Installing weak dependencies:
[Buildah]  python2-pip
[Buildah]             noarch 9.0.3-14.module+el8.1.0+3446+c3d52da3  ubi-8-appstream 2.0 M
[Buildah]  python2-setuptools
[Buildah]             noarch 39.0.1-11.module+el8.1.0+3446+c3d52da3 ubi-8-appstream 643 k
[Buildah] Enabling module streams:
[Buildah]  python27          2.7                                                         
[Buildah] 
[Buildah] Transaction Summary
[Buildah] ================================================================================
[Buildah] Install  6 Packages
[Buildah] 
[Buildah] Total download size: 10 M
[Buildah] Installed size: 39 M
[Buildah] Downloading Packages:
[Buildah] (1/6): python2-2.7.16-12.module+el8.1.0+4148+33 314 kB/s | 109 kB     00:00    
[Buildah] (2/6): python2-setuptools-wheel-39.0.1-11.modul 543 kB/s | 289 kB     00:00    
[Buildah] (3/6): python2-pip-9.0.3-14.module+el8.1.0+3446 2.8 MB/s | 2.0 MB     00:00    
[Buildah] (4/6): python2-libs-2.7.16-12.module+el8.1.0+41 4.3 MB/s | 6.0 MB     00:01    
[Buildah] (5/6): python2-setuptools-39.0.1-11.module+el8. 672 kB/s | 643 kB     00:00    
[Buildah] (6/6): python2-pip-wheel-9.0.3-14.module+el8.1. 504 kB/s | 1.2 MB     00:02    
[Buildah] --------------------------------------------------------------------------------
[Buildah] Total                                           2.9 MB/s |  10 MB     00:03     
[Buildah] Running transaction check
[Buildah] Transaction check succeeded.
[Buildah] Running transaction test
[Buildah] Transaction test succeeded.
[Buildah] Running transaction
[Buildah]   Preparing        :                                                        1/1 
[Buildah]   Installing       : python2-pip-wheel-9.0.3-14.module+el8.1.0+3446+c3d52   1/6 
[Buildah]   Installing       : python2-setuptools-wheel-39.0.1-11.module+el8.1.0+34   2/6 
[Buildah]   Installing       : python2-libs-2.7.16-12.module+el8.1.0+4148+33a50073.   3/6 
[Buildah]   Installing       : python2-pip-9.0.3-14.module+el8.1.0+3446+c3d52da3.no   4/6 
[Buildah]   Installing       : python2-setuptools-39.0.1-11.module+el8.1.0+3446+c3d   5/6 
[Buildah]   Installing       : python2-2.7.16-12.module+el8.1.0+4148+33a50073.x86_6   6/6 
[Buildah]   Running scriptlet: python2-2.7.16-12.module+el8.1.0+4148+33a50073.x86_6   6/6 
[Buildah]   Verifying        : python2-2.7.16-12.module+el8.1.0+4148+33a50073.x86_6   1/6 
[Buildah]   Verifying        : python2-libs-2.7.16-12.module+el8.1.0+4148+33a50073.   2/6 
[Buildah]   Verifying        : python2-setuptools-wheel-39.0.1-11.module+el8.1.0+34   3/6 
[Buildah]   Verifying        : python2-pip-9.0.3-14.module+el8.1.0+3446+c3d52da3.no   4/6 
[Buildah]   Verifying        : python2-setuptools-39.0.1-11.module+el8.1.0+3446+c3d   5/6 
[Buildah]   Verifying        : python2-pip-wheel-9.0.3-14.module+el8.1.0+3446+c3d52   6/6 
[Buildah] Installed products updated.
[Buildah] 
[Buildah] Installed:
[Buildah]   python2-2.7.16-12.module+el8.1.0+4148+33a50073.x86_64                         
[Buildah]   python2-pip-9.0.3-14.module+el8.1.0+3446+c3d52da3.noarch                      
[Buildah]   python2-setuptools-39.0.1-11.module+el8.1.0+3446+c3d52da3.noarch              
[Buildah]   python2-libs-2.7.16-12.module+el8.1.0+4148+33a50073.x86_64                    
[Buildah]   python2-setuptools-wheel-39.0.1-11.module+el8.1.0+3446+c3d52da3.noarch        
[Buildah]   python2-pip-wheel-9.0.3-14.module+el8.1.0+3446+c3d52da3.noarch                
[Buildah] 
[Buildah] Complete!
[Buildah] 0 files removed
[Buildah] useradd: /etc/passwd.323: lock file already used
[Buildah] useradd: cannot lock /etc/passwd; try again later.
[Buildah] subprocess exited with status 1
[Buildah] subprocess exited with status 1
[Buildah] error building at STEP "RUN yum upgrade --disableplugin=subscription-manager -y  && yum install --disableplugin=subscription-manager python2 -y  && yum clean --disableplugin=subscription-manager packages  && ln -s /usr/bin/python2 /usr/bin/python  && useradd --uid 1000 --gid 0 --shell /bin/bash --create-home node": exit status 1
[Error] exit status 1

Also i tested for another pipeline java-mp build push in this CRC container and for this pipeline the pipelinerun was successfull however it took below time

Pipeline(build-push-pipeline)	Buildah image	Time
java-mp	gireeshpunathil/appsody-buildah:v1	11min
--	--	--
java-mp	appsody/appsody-buildah:0.5.6-buildah1.9.0	13min

kabanero-io / kabanero-pipelines

Bug: performance problem in build-bud step of nodejs and nodejs loopback pipeline #121