containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.35k stars 774 forks source link

Strange behaviour with versions above v1.21.4 #4704

Open starkmatt opened 1 year ago

starkmatt commented 1 year ago

Description There's been some odd issues observed with Buildah versions above v1.21.4 with regards to Java applications that are built in the container. Currently I'm using open-liberty-jdk container, and when we've recently updated our Helpers to version v1.28.0 all sorts of issues started occurring specifically with Control-M jobs.

The error would alternate between a Port issue ?

info:    could NOT find free port, search range 7625-8625
info:    no free port was found
info:    could NOT find free port, search range 7722-8722
info:    no free port was foundFailed to deploy new image
error:   java.net.BindException: no free port was found
error:   
info:    exiting provision process with exit code (0-ok):6
error:   provision image failed

And Hostname issue

info:    Agent unique physical name must be shorter then 86 characters[ Automated installation failed! ]
Installation has failed!
For more details refer to :
/home/ctmag/BMCINSTALL/log/BMC_Control-M_Agent_Install_2023.02.23.12.53.11.log
info:    Failed to deploy new image
error:   org.apache.commons.exec.ExecuteException: return error for product DRKAI.9.0.20.200_Linux-x86_64.tar.Z-DEPLOY: using command /home/ctmag/.ctmTemp/DRKAI.9.0.20.200_Linux-x86_64.tar/setup.sh -silent /home/ctmag/.ctmTemp/agent.linux.xml. rc=1 (Exit value: 1)
error:   
info:    exiting provision process with exit code (0-ok):6
error:   provision image failed

Went through the usual dance of debugging ports and hostname and noticed that those were not the case at all. Hostname was only 16 characters and the relative ports were open and tested.

Then updated to version v1.29.0 the thinking was perhaps v1.28.0 had an issue but it was the same. After exhausting all options, decided to head back to the version that builds were working v1.21.4 and it was magic, no issues.

Steps to reproduce the issue:

  1. Docker Build with open-liberty-jdk
  2. Ctrl-M Job doing a quick provison
  3. Observe the build behaviour

Describe the results you received: Either

info:    could NOT find free port, search range 7625-8625
info:    no free port was found
info:    could NOT find free port, search range 7722-8722
info:    no free port was foundFailed to deploy new image
error:   java.net.BindException: no free port was found
error:   
info:    exiting provision process with exit code (0-ok):6
error:   provision image failed

Or

info:    Agent unique physical name must be shorter then 86 characters[ Automated installation failed! ]
Installation has failed!
For more details refer to :
/home/ctmag/BMCINSTALL/log/BMC_Control-M_Agent_Install_2023.02.23.12.53.11.log
info:    Failed to deploy new image
error:   org.apache.commons.exec.ExecuteException: return error for product DRKAI.9.0.20.200_Linux-x86_64.tar.Z-DEPLOY: using command /home/ctmag/.ctmTemp/DRKAI.9.0.20.200_Linux-x86_64.tar/setup.sh -silent /home/ctmag/.ctmTemp/agent.linux.xml. rc=1 (Exit value: 1)
error:   
info:    exiting provision process with exit code (0-ok):6
error:   provision image failed

Describe the results you expected:

info:    Killing Control-M/Agent Listener pid:6129
info:    2023-04-04 10:26:32 Listener process stopped
info:    Killing Control-M/Agent Tracker pid:6217
info:    2023-04-04 10:26:33 Tracker process stoppedKilling Control-M/Agent Agent Java Process pid:6041
info:    1 seconds - 6041 is still alive
info:    2 seconds - 6041 is still alive
info:    3 seconds - 6041 is still alive
info:    4 seconds - 6041 is still alive
info:    5 seconds - 6041 is still alive
info:    6 seconds - 6041 is still alive
info:    7 seconds - 6041 is still alive
info:    8 seconds - 6041 is still alive
info:    9 seconds - 6041 is still alive
info:    10 seconds - 6041 is still alive
info:    11 seconds - 6041 is still alive
info:    12 seconds - 6041 is still alive
info:    13 seconds - 6041 is still alive
info:    14 seconds - 6041 is still alive
info:    15 seconds - 6041 is still aliveAttempting brutal stop...
2023-04-04 10:26:48 Agent Java Process process was brutaly stopped
info:    2023-04-04 10:26:49 Agent Java Process process stopped
info:    provisioning image Agent_20.Linux ended successfully
info:    exiting provision process with exit code (0-ok):0
info:    provision image succeeded
installation ended successfully

Output of buildah version:

Any versions above v1.21.4 had the issue.

I wanted to bring this to your attention for some reflections and perhaps some discussions on matter ? All builds are running within Pipelines in GitLab not doing any local builds at all.

Current workarounds for builds that are using Ctrl-M Provisioning is to set the variable of the buildah version to v1.21.4 no other type of builds presented these symptoms thus far.

Thanks for reading! 🚀 🙏

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.