`archive.extracted` fails for `.img.xz`

brianthelion commented 5 years ago

Description of Issue

A very vanilla archive.extracted state is failing for XZ archives.

First, with:

    ID: /tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507
                                Function: archive.extracted
                                  Result: False
                                 Comment: Could not guess archive_format from the value of the 'source' argument. Please set this archive_format to one of the following: tar, rar, zip
                                 Started: 17:52:20.209757
                                Duration: 4.098 ms
                                 Changes:

And then with archive_type: zip set:

ID: /tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507
                                Function: archive.extracted
                                  Result: False
                                 Comment: /var/cache/salt/minion/extrn_files/api/cdimage.ubuntu.com/releases/bionic/release/ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz is not a ZIP file. Additional info follows:

                                          archive location:
                                              /var/cache/salt/minion/extrn_files/api/cdimage.ubuntu.com/releases/bionic/release/ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz

                                          The following workarounds must be used for this state to proceed (assuming the source file is a valid zip archive):

                                          - 'if_missing' must be set
                                 Started: 17:54:50.135402
                                Duration: 2323.976 ms
                                 Changes:

Setup

Run the bellow:

# unzip_state.sls
various_zip_packages:
  pkg.installed:
    - pkgs:
        - unzip
        - xz-utils

/tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507:
  archive.extracted:
    - source: http://cdimage.ubuntu.com/releases/bionic/release/ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz
    - source_hash: 0b29297cf9f2a2112d634b1e29f94507
    - archive_format: zip
    - require:
        - various_zip_packages

Steps to Reproduce Issue

Versions Report

Salt Version:
           Salt: 2019.2.2-3-g71e0088

Dependency Versions:
           cffi: 1.13.1
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Oct  7 2019, 12:59:55)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 18.1.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-65-generic
         system: Linux
        version: Ubuntu 18.04 bionic

frogunder commented 5 years ago

@brianthelion Thank you for reporting this issue.

I am able to replicate it.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

nergdron commented 4 years ago

I just ran into this on 3001rc1, and it occurs even when I specify archive_format: tar. any ETA on getting this fixed?

boltronics commented 4 years ago

The problem here is that ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz is not an "archive" file, such as the tar, zip, rar or cpio archive formats (although even cpio doesn't seem to be supported). Instead, it would only extract to a single file - ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img. The image file may contain many files within, but that's only because you can mount it once extracted (since will probably contain a file system), but this is not a true archive format. Essentially it would be treated as a single file that just happens to be compressed.

I'm facing a similar issue. I was hoping I could use archive.extracted to extract /usr/share/doc/rsync/scripts/rrsync.gz, but that too isn't an archive file - it's just a single file (the rrsync Perl script) that is gzip-compressed. If it were rrsync.tar.gz, that would be a different story - but wrapping a single file into a tar "archive" would be redundant. Archives are only there to join multiple files together, and I only have a single file.

Perhaps if Salt's archive module were to be renamed "compression", adding this support would make more sense - and I would like to see this support added in some form. Alternatively, it might make more sense to just add the required new functionality into a separate module.

Browsing through the source code, I see xz-compressed archives (ie. .tar.xz files) are supported. Typically xz-compression would be used with tar archives to compress multiple files, since rar and zip archive formats have compression support built-in. By contrast, tar and cpio are pure archive formats and offer no compression, so need to be used with a separate compression library such as those for xz, bzip2, gzip, etc. This is because *NIX tools will typically follow the tradition of only doing one thing, and doing it well.

Probably this should be marked as not-a-bug/invalid, or alternatively marked as a feature request. In the meantime, you might want to use file.managed to fetch the file, and then something like this:

Extract /tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507:
  cmd.run:
    - name: zcat /path/to/ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz > /tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507
    - unless: test -s /tmp/vendor-images/ubuntu-18.04.3-preinstalled-server-arm64+raspi3/0b29297cf9f2a2112d634b1e29f94507
    - require:
      - file: Fetch ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img.xz

saltstack / salt