Assigned an extent beyond the ISO

tkontogi commented 1 year ago

Summary

Hello,

Using the playbook listed in "Steps to Reproduce" I get the failure listed in "Actual Results"

Thank you

Issue Type

Bug Report

Component Name

community.general.iso_customize

Ansible Version

$ ansible --version
ansible [core 2.14.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/tkonto/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/tkonto/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general
# /home/tkonto/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 6.1.0  

# /usr/lib/python3.11/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 6.0.1

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = /etc/ansible/ansible.cfg

OS / Environment

Fedora 37

Steps to Reproduce


---
- name: ISO file related tasks
  hosts: all
  connection: local
  gather_facts: false
  become: false
  collections:
   - community.general
  vars:

  tasks:

  - name: "Inject the kickstart file inside the ISO file"
    community.general.iso_customize:
      src_iso: "/home/tkonto/projects/code/VEG/test/VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso"
      dest_iso: "/home/tkonto/projects/code/VEG/test/VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.ALTERED.iso"
      add_files:
        - src_file: "/home/tkonto/projects/code/VEG/test/TEST.FILE"
          dest_file: "/TEST.FILE"
    register: customize_iso_result

Expected Results

I expect the TEST.FILE to be injected in the iso file and have a new iso file created

Actual Results

[tkonto@tweety-lap test]$ ansible-playbook -i localhost, ./iso.yml -vvvv
ansible-playbook [core 2.14.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/tkonto/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/tkonto/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Set default localhost to localhost
Parsed localhost, inventory source with host_list plugin
Loading collection community.general from /home/tkonto/.ansible/collections/ansible_collections/community/general
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.11/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: iso.yml ****************************************************************************
Positional arguments: ./iso.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('localhost,',)
forks: 5
1 plays in ./iso.yml

PLAY [ISO file related tasks] ****************************************************************

TASK [Inject the kickstart file inside the ISO file] *****************************************
task path: /home/tkonto/projects/code/VodafoneEG/test/iso.yml:13
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: tkonto
<localhost> EXEC /bin/sh -c 'echo ~tkonto && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tkonto/.ansible/tmp `"&& mkdir "` echo /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674 `" && echo ansible-tmp-1671186561.127521-20933-202705775332674="` echo /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674 `" ) && sleep 0'
<localhost> Attempting python interpreter discovery
<localhost> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'python3.11'"'"'; command -v '"'"'python3.10'"'"'; command -v '"'"'python3.9'"'"'; command -v '"'"'python3.8'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 && sleep 0'
Using module file /home/tkonto/.ansible/collections/ansible_collections/community/general/plugins/modules/iso_customize.py
<localhost> PUT /home/tkonto/.ansible/tmp/ansible-local-20929okcr9g61/tmpn2bykiq3 TO /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674/AnsiballZ_iso_customize.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674/ /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674/AnsiballZ_iso_customize.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674/AnsiballZ_iso_customize.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/tkonto/.ansible/tmp/ansible-tmp-1671186561.127521-20933-202705775332674/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_community.general.iso_customize_payload_rx4v4squ/ansible_community.general.iso_customize_payload.zip/ansible_collections/community/general/plugins/modules/iso_customize.py", line 209, in iso_add_file
  File "/usr/lib/python3.11/site-packages/pycdlib/pycdlib.py", line 4423, in add_file
    self._finish_add(0, num_bytes_to_add)
  File "/usr/lib/python3.11/site-packages/pycdlib/pycdlib.py", line 3076, in _finish_add
    self._reshuffle_extents()
  File "/usr/lib/python3.11/site-packages/pycdlib/pycdlib.py", line 1665, in _reshuffle_extents
    raise pycdlibexception.PyCdlibInternalError('Assigned an extent beyond the ISO (%d > %d)' % (current_extent, self.pvd.space_size))
fatal: [localhost]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "add_files": [
                {
                    "dest_file": "/TEST.FILE",
                    "src_file": "/home/tkonto/projects/code/VEG/test/TEST.FILE"
                }
            ],
            "delete_files": [],
            "dest_iso": "/home/tkonto/projects/code/VEG/test/VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.ALTERED.iso",
            "src_iso": "/home/tkonto/projects/code/VEG/test/VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso"
        }
    },
    "msg": "Failed to add local file /home/tkonto/projects/code/VEG/test/TEST.FILE to ISO with error: Assigned an extent beyond the ISO (204214 > 204212)"
}

PLAY RECAP ***********************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

tkontogi commented 1 year ago

[tkonto@tweety-lap test]$ ls -l VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso 
-rw-r--r--. 1 tkonto tkonto 418224128 Dec 15 18:16 VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso

[tkonto@tweety-lap test]$ file VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso 
VMware_ESXi_6.7.0_17700523_HPE_Gen9plus_670.U3.10.7.0.132_May2021.iso: ISO 9660 CD-ROM filesystem data 'HPE-ESXI-6.7.0-UPDATE3-ISO-GEN9P' (bootable)

tkontogi commented 1 year ago

[tkonto@tweety-lap test]$ file TEST.FILE 
TEST.FILE: ASCII text

[tkonto@tweety-lap test]$ ls -l TEST.FILE 
-rw-r--r--. 1 tkonto tkonto 35 Dec 16 12:51 TEST.FILE

clalancette commented 1 year ago

Sorry for the long delay here. In short, I can reproduce this problem (outside of ansible, which makes it easier). There must be something about the structure of that particular ISO that is different than pycdlib expects. I'll take a closer look; thanks for the report.

clalancette commented 1 year ago

All right, I see what is happening. In all other ISOs I've looked at until this point, the Path Table is always created in groups of 2 extents (4096 bytes). There is both a little-endian and big-endian Path Table on an ISO, so in simple ISOs this takes up 4 extents (8192 bytes).

However, the standard doesn't mandate that, and in fact the VMWare ISO has Path Tables with only 1 extent (2048 bytes). With the little-endian and big-endian ones, the total is 2 extents (4096 bytes).

Because I've never seen an ISO like this before, pycdlib has an assumption about the 2 extents per Path Table thing, and that is what is being violated here. I'm not quite sure how to fix this yet, but at least I understand the problem now.

clalancette commented 1 year ago

All right, 692a2dd9736acc8650ad065d4ef8264d0f446df1 should fix this particular problem. I don't love the solution, but I also don't see another way to fix it right now; we just don't have the required information. Please give this a shot when you get a chance. I'm going to close this out for now.

clalancette / pycdlib