shanemcd
commented
2 years ago @AlanCoding @pabelanger Does this look familiar to either of you? Is this a bug in the assemble script?
Open wohnout opened 2 years ago
shanemcd
commented
2 years ago @AlanCoding @pabelanger Does this look familiar to either of you? Is this a bug in the assemble script?
AlanCoding
commented
2 years ago This error looks like it's in the install script, as opposed to the assemble script
https://github.com/ansible/python-builder-image/blob/main/scripts/install-from-bindep#L63
But right before it sets CONSTRAINTS="-c /output/upper-constraints.txt", it checks if that file exists. So it's not at all obvious how this could happen. I don't think builder or the dockerfile messes with the CONSTRAINTS env var, so I really don't know.
aimcod
commented
2 years ago I am having the same issue:
Running setup.py install for vSphere-Automation-SDK: started
Running setup.py install for vSphere-Automation-SDK: finished with status 'done'
Successfully installed PyJWT-2.3.0 adal-1.2.7 aiohttp-3.8.1 aiosignal-1.2.0 ansible-pylibssh-1.0.0 appdirs-1.4.4 async-timeout-4.0.2 attrs-22.1.0 awxkit-21.6.0 azure-common-1.1.28 azure-core-1.25.1 azure-mgmt-compute-27.2.0 azure-mgmt-core-1.3.2 azure-mgmt-network-21.0.1 azure-mgmt-resource-21.1.0 azure-mgmt-storage-20.1.0 boto3-1.24.76 botocore-1.27.76 cachetools-5.2.0 certifi-2022.9.14 charset-normalizer-2.1.1 datetime-4.7 dnspython-2.2.1 docker-6.0.0 dogpile.cache-1.1.8 enum34-1.1.10 frozenlist-1.3.1 future-0.18.2 google-auth-2.11.1 grpcio-1.48.1 ipaddress-1.0.23 iso8601-1.0.2 isodate-0.6.1 jmespath-1.0.1 jsonpatch-1.32 jsonpointer-2.3 jsonschema-3.2.0 jxmlease-1.0.3 keystoneauth1-5.0.0 kubernetes-24.2.0 msrest-0.7.1 msrestazure-0.6.4 multidict-6.0.2 munch-2.5.0 netaddr-0.8.0 netapp-lib-2021.6.25 netifaces-0.11.0 nsx-policy-python-sdk-4.0.0.0.0 nsx-python-sdk-4.0.0.0.0 nsx-vmc-aws-integration-python-sdk-4.0.0.0.0 nsx-vmc-policy-python-sdk-4.0.0.0.0 oauthlib-3.2.1 objectpath-0.6.1 omsdk-1.2.490 openstacksdk-0.61.0 ordereddict-1.1 os-service-types-1.7.0 paramiko-2.7.1 pbr-5.10.0 protobuf-4.21.6 purefusion-1.0.4 purity-fb-1.12.3 py-pure-client-1.14.0 pyVmomi-7.0.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycryptodomex-3.15.0 pynetbox-6.6.2 pyrsistent-0.18.1 pysmi-0.3.4 pysnmp-4.4.12 pysnmp-mibs-0.1.6 python-dateutil-2.8.2 requests-2.28.1 requests-oauthlib-1.3.1 requestsexceptions-1.4.0 rsa-4.9 s3transfer-0.6.0 simplejson-3.17.6 solidfire-sdk-python-12.3.0.203 stevedore-4.0.0 textfsm-1.1.3 ttp-0.9.1 typing-3.7.4.3 typing-extensions-4.3.0 urllib3-1.26.12 vSphere-Automation-SDK-1.78.0 vapi-client-bindings-3.9.0 vapi-common-client-2.34.0 vapi-runtime-2.34.0 vmc-client-bindings-1.60.0 vmc-draas-client-bindings-1.19.0 websocket-client-1.4.1 yarl-1.8.1 zabbix-api-0.5.4 zope.interface-5.4.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
+ EXTRAS=
+ '[' -f /output/packages.txt ']'
++ wc -l
++ ls -1 /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
+ '[' 2 -gt 0 ']'
+ pip3 uninstall -y /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
Found existing installation: ansible-core 2.12.5.post0
Uninstalling ansible-core-2.12.5.post0:
Successfully uninstalled ansible-core-2.12.5.post0
Found existing installation: ansible-runner 2.1.4.dev2
Uninstalling ansible-runner-2.1.4.dev2:
Successfully uninstalled ansible-runner-2.1.4.dev2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ pip3 install -c /output/upper-constraints.txt --cache-dir=/output/wheels /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/output/upper-constraints.txt'
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
Error: error building at STEP "RUN /output/install-from-bindep && rm -rf /output/wheels": error while running runtime: exit status 1I was able to successfully build an image before. Now I am not.
All that changed between builds was the requirements.yml contents.
So not quite the issue OP has but the same result.
What I did was remove _build as well as the Containerfile and re-ran the build.
In my case, that worked.
Attaching the next run after removing the older files.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
+ EXTRAS=
+ '[' -f /output/packages.txt ']'
++ wc -l
++ ls -1 /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
+ '[' 2 -gt 0 ']'
+ pip3 uninstall -y /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
Found existing installation: ansible-core 2.12.5.post0
Uninstalling ansible-core-2.12.5.post0:
Successfully uninstalled ansible-core-2.12.5.post0
Found existing installation: ansible-runner 2.1.4.dev2
Uninstalling ansible-runner-2.1.4.dev2:
Successfully uninstalled ansible-runner-2.1.4.dev2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
**+ pip3 install -c /output/upper-constraints.txt --cache-dir=/output/wheels /output/wheels/ansible_core-2.12.5.post0-py3-**none-any.whl /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
Processing /output/wheels/ansible_core-2.12.5.post0-py3-none-any.whl
Processing /output/wheels/ansible_runner-2.1.4.dev2-py3-none-any.whl
Requirement already satisfied: PyYAML in /usr/lib64/python3.8/site-packages (from ansible-core==2.12.5.post0) (5.4.1)
**Requirement already satisfied: resolvelib<0.6.0,>=0.5.3 in /usr/local/lib/python3.8/site-packages (from ansible-core==2.12.5.post0) (0.5.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/site-packages (from ansible-core==2.12.5.post0) (21.3)
Requirement already satisfied: cryptography in /usr/lib64/python3.8/site-packages (from ansible-core==2.12.5.post0) (2.8)**
Here's my execution-environment.yaml file:
---
version: 1
dependencies:
galaxy: requirements.yml
python: requirements.txt
system: bindep.txt
additional_build_steps:
append:
- RUN yum upgrade -y
- RUN yum install -y python39
- RUN yum install -y python39-pip
- RUN yum install -y krb5-devel
- RUN yum install -y python39-devel
- RUN yum install -y gcc
- RUN alternatives --set python /usr/bin/python3.9
- RUN pip3.9 install --upgrade --force pip
- RUN python -m pip install --upgrade pip
- RUN pip uninstall --yes ansible-runner && pip install ansible-runner==2.1.4
- RUN pip install pypsrp[kerberos]
- COPY --from=quay.io/project-receptor/receptor:latest /usr/bin/receptor /usr/bin/receptor
- RUN mkdir -p /var/run/receptor
- USER 1000
- ADD run.sh /run.sh
- CMD /run.sh
- RUN git lfs installand requirements.yml file
collections:
- name: amazon.aws
- name: ansible.netcommon
- name: ansible.posix
- name: ansible.utils
- name: ansible.windows
- name: arista.eos
- name: awx.awx
- name: chocolatey.chocolatey
- name: community.aws
- name: community.crypto
- name: community.dns
- name: community.docker
- name: community.general
- name: community.grafana
- name: community.libvirt
- name: community.mongodb
- name: community.mysql
- name: community.network
- name: community.okd
- name: community.postgresql
- name: community.proxysql
- name: community.rabbitmq
- name: community.vmware
- name: community.windows
- name: community.zabbix
- name: containers.podman
- name: dellemc.enterprise_sonic
- name: dellemc.openmanage
- name: dellemc.os10
- name: dellemc.os6
- name: dellemc.os9
- name: f5networks.f5_modules
- name: frr.frr
- name: gluster.gluster
- name: kubernetes.core
- name: netapp.aws
- name: netapp.cloudmanager
- name: netapp.elementsw
- name: netapp.ontap
- name: netapp.storagegrid
- name: netapp_eseries.santricity
- name: netbox.netbox
- name: openstack.cloud
- name: purestorage.flasharray
- name: purestorage.flashblade
- name: purestorage.fusion
- name: theforeman.foreman
- name: vmware.vmware_restrequirements.txt and bindep.txt for measure:
[sysadmin@dev-ansible-02 ansible-builder]$ cat requirements.txt bindep.txt
awxkit==21.6.0
urllib3
python38-devel [platform:rpm compile]
subversion [platform:rpm]
subversion [platform:dpkg]
git-lfs [platform:rpm]I hope this helps.
smullenrga
commented
2 years ago I have been having this install-from-bindep error using ansible-builder ina playbook that worked a few months ago. I was concentrating on the bindep.txt because it says install-from-bindep has failed. After digging in and adding some debug code to ansible-builder code, I'd identified that the actual docker command ended up being "docker build -f /tmp/eebuild/Dockerfile -t
So, just noting that A: getting the exact docker command and manually running it may help and B: install-from-bindep errors may actually be python pip install errors.
williamhanseneric
commented
1 year ago I have this same issue, this worked a few months ago and now it is failing for unknown reasons using the process I documented previous.
The failure is, {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PKGMGR_OPTS=--nodocs --setopt=install_weak_deps=0 --setopt=rhel-8-for-x86_64-appstream-rpms.excludepkgs=ansible-core HOME=/home/runner DESCRIPTION=Red Hat Ansible Automation Platform Minimal Execution Environment container=oci] Command:run Args:[/output/install-from-bindep && rm -rf /output/wheels] Flags:[] Attrs:map[] Message:RUN /output/install-from-bindep && rm -rf /output/wheels Original:RUN /output/install-from-bindep && rm -rf /output/wheels}: error while running runtime: exit status 1 Error: error building at STEP "RUN /output/install-from-bindep && rm -rf /output/wheels": error while running runtime: exit status 1
Beforehand, earlier in the log, it refers to install-from-bindep with this "too many arguments" error on line 28.
I literally have nothing inside of my execution_environment.yml for the container I am builder, except for an ansible config file. I don't have any python, collection, or binary requirements. But, I still receive this error.
Is the error coming from the Ansible builder container? This is also fairly "stock" for us.
williamhanseneric
commented
1 year ago Here is the source of the error, from within the container
bash-4.4# /usr/bin/microdnf update -y --nodocs --setopt=install_weak_deps=0 --setopt=rhel-8-for-x86_64-appstream-rpms.excludepkgs=ansible-core
Downloading metadata...
Package Repository Size
Upgrading:
util-linux-2.32.1-39.el8_7.x86_64 rhel-8-for-x86_64-ba 2.6 MB
replacing util-linux-2.32.1-38.el8.x86_64
Transaction Summary:
Installing: 0 packages
Reinstalling: 0 packages
Upgrading: 1 packages
Obsoleting: 0 packages
Removing: 0 packages
Downgrading: 0 packages
Running transaction test...
Updating: util-linux;2.32.1-39.el8_7;x86_64;rhel-8-for-x86_64-baseos-rpms
error: Error -1 running transaction
bash-4.4#
aimcod
commented
1 year ago @AlanCoding @shanemcd @wohnout
I ran into this again, but this time, deleting the _build folder and Container file did nothing.
I investigated a bit and concluded that the error is being caused by the missing /output/upper-constraints.txt file, which is totally optional.
I don't know why the conditional check succeeds, since neither the builder, nor the runner images contain that file(although the builder is the source?) but this can be easily avoided with a change in syntax from
# If there's a constraints file, use it.
if [ -f /output/upper-constraints.txt ] ; then
CONSTRAINTS="-c /output/upper-constraints.txt"
fi
to
# If there's a constraints file, use it.
if [ -f /output/upper-constraints.txt ] ; then
CONSTRAINTS="-c /output/upper-constraints.txt"
else
CONSTRAINTS=""
fiOr by allowing us to define the constraints file ourselves, as a dependency, maybe? Should we do that, but is it just not documented?
Or by adding an empty /output/upper-constraints.txt file, which is what I ended up doing.
This allowed the installation to continue without any errors, as I effectively did not pass any constraints to the pip commands and I was finally, after, 3 days of work on this(yeah, I fumbled a lot), able to get an image ready for our AWX Env, with the new collections installed.
Now, the important bit is that I did not use ansible-builder, as ansible-builder rewrites the Containerfile every time it runs.
Instead I ran podman build with the following Containerfile.
ARG EE_BASE_IMAGE=bec0dc171168 #this is the local image ID for the ansible-runner image
ARG EE_BUILDER_IMAGE=b0348faa7f41 #this is the local image ID for the ansible-builder image
FROM $EE_BASE_IMAGE as galaxy
ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
ARG ANSIBLE_GALAXY_CLI_ROLE_OPTS=
USER root
ADD _build /build
WORKDIR /build
RUN ansible-galaxy role install $ANSIBLE_GALAXY_CLI_ROLE_OPTS -r requirements.yml --roles-path "/usr/share/ansible/roles"
RUN ANSIBLE_GALAXY_DISABLE_GPG_VERIFY=1 ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path "/usr/share/ansible/collections"
FROM $EE_BUILDER_IMAGE as builder
COPY --from=galaxy /usr/share/ansible /usr/share/ansible
ADD _build/requirements.txt requirements.txt
ADD _build/bindep.txt bindep.txt
RUN pip install ansible-builder assemble #for some reason, these packages are missing, when running podman build instead of ansible-builder, so they need to be manually installed.
RUN ansible-builder introspect --sanitize --user-pip=requirements.txt --user-bindep=bindep.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
RUN assemble
FROM $EE_BASE_IMAGE
USER root
COPY --from=galaxy /usr/share/ansible /usr/share/ansible
COPY --from=builder /output/ /output/
RUN touch /output/upper-constraints.txt #this creates an empty /output/upper-constraints.txt file, so that we avoid the error.
RUN /output/install-from-bindep && rm -rf /output/wheels
RUN yum upgrade -y
RUN yum install -y python39
RUN yum install -y python39-pip
RUN yum install -y krb5-devel
RUN yum install -y python39-devel python39-setuptools
RUN yum install -y gcc
RUN yum install -y libcurl-devel openssl-devel git-lfs
RUN alternatives --set python /usr/bin/python3.9
RUN pip3.9 install --upgrade --force pip
RUN python3.9 -m pip install --target=/usr/lib/python3.9/site-packages/ pywinrm --upgrade
RUN python3.9 -m pip install --target=/usr/lib/python3.9/site-packages/ pywinrm[kerberos] --upgrade
RUN python -m pip install --upgrade pip
RUN pip3 uninstall --yes ansible-runner && pip3 install ansible-runner>=2.2.1
RUN pip3 install setuptools-rust
RUN pip3 install pypsrp[kerberos]
RUN pip3 install pyVim PyVmomi
RUN pip3 install passlib
RUN pip3 install botocore boto3 awscli
RUN pip3 install py-zabbix azure-mgmt-resource python-dotenv pycurl
RUN python -m pip install ansible ansible-core
COPY --from=quay.io/project-receptor/receptor:latest /usr/bin/receptor /usr/bin/receptor
RUN mkdir -p /var/run/receptor
USER 1000
ADD run.sh /run.sh
CMD /run.sh
RUN git lfs install
LABEL ansible-execution-environment=trueI have added a comment next to each modification I had to make.
Perhaps ansible-builder could allow us to pass some flags that allow the skipping of the Containerfile/Dockerfile recreation.
I am trying to build custom EE with different version of ansible. I am able to build custom EE with different set of collections or bindep with command
ansible-builder build --tag xxx.azurecr.io/ansible/awx-custom-ee:2.11-65fd33e7f6f2 --context ./ --container-runtime podmanbut if add build-arg EE_BASE_IMAGE like this
ansible-builder build --tag xxx.azurecr.io/ansible/awx-custom-ee:2.11-65fd33e7f6f2 --context ./ --container-runtime podman --build-arg EE_BASE_IMAGE=quay.io/ansible/ansible-runner:stable-2.11-latestIt fails with error:
When I do compare between these two outputs it thinks that this file is present
I was able to build same way custom EE few months ago but I am not able anymore.
Can you please help me what I am doing wrong?