kubernetes-sigs / image-builder

Tools for building Kubernetes disk images
https://image-builder.sigs.k8s.io/
Apache License 2.0
402 stars 394 forks source link

(proxmox) after a seemingly successful build, in the finishing steps ansible error #1591

Closed lknite closed 1 month ago

lknite commented 1 month ago

What steps did you take and what happened?

[~/g/image-builder (main)]
[travis@wb]$ cat go.sh 
#!/bin/bash

# configure via env vars
export PACKER_LOG=1
export PROXMOX_BRIDGE="vmbr0"
export PROXMOX_ISO_POOL="tower"
export PROXMOX_NODE="pve-c"
export PROXMOX_STORAGE_POOL="cephfs"
export PROXMOX_TOKEN="..redacted.."
export PROXMOX_URL="https://10.0.0.21:8006/api2/json"
export PROXMOX_USERNAME="capmox@pve!capi"

# update dependencies
cd images/capi
make deps-proxmox
cd ../..

# build images
cd images/capi
#make help
make build-proxmox-ubuntu-2204
#make build-proxmox-ubuntu-2404
cd ../..

What did you expect to happen?

image-builder to complete successfully.

Relevant log output

2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 SSH proxy: accepted connection
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 authentication attempt from 127.0.0.1:36004 to 127.0.0.1:44157 as builder using none
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 authentication attempt from 127.0.0.1:36004 to 127.0.0.1:44157 as builder using publickey
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 new exec request: /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/.ansible `"&& mkdir "` echo /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330 `" && echo ansible-tmp-1728866753.3781035-706042-50794845410330="` echo /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330 `" ) && sleep 0'
2024/10/13 18:45:53 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [DEBUG] Opening new ssh session
2024/10/13 18:45:53 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [DEBUG] starting remote command: /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/.ansible `"&& mkdir "` echo /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330 `" && echo ansible-tmp-1728866753.3781035-706042-50794845410330="` echo /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330 `" ) && sleep 0'
2024/10/13 18:45:53 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:53 [INFO] 120 bytes written for 'stdout'
2024/10/13 18:45:53 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:53 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:53 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:53 [INFO] 0 bytes written for 'stdin'
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [INFO] 120 bytes written for 'stdout'
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [INFO] 0 bytes written for 'stdin'
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 SSH proxy: accepted connection
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 authentication attempt from 127.0.0.1:36010 to 127.0.0.1:44157 as builder using none
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 authentication attempt from 127.0.0.1:36010 to 127.0.0.1:44157 as builder using publickey
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 new subsystem request: sftp
2024/10/13 18:45:53 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 starting sftp subsystem
2024/10/13 18:45:53 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [DEBUG] Opening new ssh session
2024/10/13 18:45:53 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:53 [DEBUG] starting remote command: /usr/lib/sftp-server -e
2024/10/13 18:45:54 [INFO] 297918 bytes written for 'stdin'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 297918 bytes written for 'stdin'
2024/10/13 18:45:54 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:54 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:54 [INFO] 669 bytes written for 'stdout'
2024/10/13 18:45:54 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:54 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 669 bytes written for 'stdout'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 SSH proxy: accepted connection
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 authentication attempt from 127.0.0.1:36014 to 127.0.0.1:44157 as builder using none
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 authentication attempt from 127.0.0.1:36014 to 127.0.0.1:44157 as builder using publickey
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 new exec request: /bin/sh -c 'rm -f -r /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330/ > /dev/null 2>&1 && sleep 0'
2024/10/13 18:45:54 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [DEBUG] Opening new ssh session
2024/10/13 18:45:54 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [DEBUG] starting remote command: /bin/sh -c 'rm -f -r /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330/ > /dev/null 2>&1 && sleep 0'
2024/10/13 18:45:54 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:54 [INFO] 0 bytes written for 'stdout'
2024/10/13 18:45:54 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:54 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:54 [INFO] RPC endpoint: Communicator ended with: 0
2024/10/13 18:45:54 [INFO] 0 bytes written for 'stdin'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 0 bytes written for 'stdout'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 0 bytes written for 'stderr'
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] RPC client: Communicator ended with: 0
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 [INFO] 0 bytes written for 'stdin'
    proxmox-iso.ubuntu-2204:
    proxmox-iso.ubuntu-2204: TASK [Gathering Facts] *********************************************************
    proxmox-iso.ubuntu-2204: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /home/travis/.ansible/tmp/ansible-local-706009rjrbsw_i/tmpffmg_xc2 /tmp/.ansible/ansible-tmp-1728866753.3781035-706042-50794845410330/AnsiballZ_setup.py:\n\n"}
    proxmox-iso.ubuntu-2204:
    proxmox-iso.ubuntu-2204: PLAY RECAP *********************************************************************
    proxmox-iso.ubuntu-2204: default                    : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
    proxmox-iso.ubuntu-2204:
2024/10/13 18:45:54 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/13 18:45:54 shutting down the SSH proxy
2024/10/13 18:45:54 [INFO] (telemetry) ending ansible
==> proxmox-iso.ubuntu-2204: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso.ubuntu-2204: Stopping VM
==> proxmox-iso.ubuntu-2204: Deleting VM
2024/10/13 18:46:00 [INFO] (telemetry) ending ubuntu-2204
==> Wait completed after 41 minutes 19 seconds
2024/10/13 18:46:00 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/10/13 18:46:00 machine readable: proxmox-iso.ubuntu-2204,error []string{"Error executing Ansible: Non-zero exit status: exit status 2"}
==> Builds finished but no artifacts were created.
Build 'proxmox-iso.ubuntu-2204' errored after 41 minutes 19 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 41 minutes 19 seconds

==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2204: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.
2024/10/13 18:46:00 [INFO] (telemetry) Finalizing.
2024/10/13 18:46:00 waiting for all plugin processes to complete...
2024/10/13 18:46:00 /home/travis/.config/packer/plugins/github.com/hashicorp/proxmox/packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64: plugin process exited
2024/10/13 18:46:00 /usr/bin/packer: plugin process exited
2024/10/13 18:46:00 /usr/bin/packer: plugin process exited
2024/10/13 18:46:00 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/13 18:46:00 /home/travis/.config/packer/plugins/github.com/YaleUniversity/goss/packer-plugin-goss_v3.2.12_x5.0_linux_amd64: plugin process exited
2024/10/13 18:46:00 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/13 18:46:00 /usr/bin/packer: plugin process exited
make: *** [Makefile:593: build-proxmox-ubuntu-2204] Error 1

Additional

/kind bug

mcbenjemaa commented 1 month ago

/kind support

lknite commented 1 month ago

If the copy is happening on my local system, then the error doesn't make sense as both those locations are definitely writable by all.

lknite commented 1 month ago

Maybe this? https://github.com/hashicorp/packer-plugin-ansible/issues/110

lknite commented 1 month ago

I'm not sure what is happening. I added the line "transfer_method = smart" to ansible.cfg and, though there was a lot of "busy" looking output, the build "appears" to have worked.

$ mkdir -p ./images/capi/downloaded_iso_path

$ cat ./images/capi/ansible.cfg | tail -n 4

[ssh_connection]
pipelining = False
transfer_method = smart

image

Going to try via the container now ...

lknite commented 1 month ago

Trying to run via container.

This looks to be working, testing. Have to pass in a folder to use as a download folder:

docker run -it --rm --network=host \
  --env-file proxmox.env \
  -v ./images/capi/downloaded_iso_path:/home/imagebuilder/images/capi/downloaded_iso_path \
  registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.37 build-proxmox-ubuntu-2204
lknite commented 1 month ago

Container based build worked without the ansible issue.

lknite commented 1 month ago

Closing as I can just use the container as a work-around.