Auto decompress images from appliances and mirrors support

[boenrobot]

Currently, the "compression" property in an appliance image entry has purely informative nature. GNS3 downloads the compressed file upon installation, and without decompressing it, renames it to what's specified in "filename", and mounts that... which ultimately fails to boot, because of the compression.

Instead, if the file is compressed, it should be decompressed after download, with the uncompressed file being mounted.

Because archives can contain multiple files, the schema should be expanded to include a way to specify the expected filename inside the archive that would ultimately be used. In its absence, the value of "filename" should be treated as the expected filename, relative from the root of the archive, with that also being used as the final file that's mounted.

e.g. (with the property being tentatively named "compression-target") instead of writing

        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "version": "15.05",
            "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
            "filesize": 55050240,
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
            "compression": "gzip"
        }

(which should do what you'd expect, except it currently doesn't)

one should be able to write

        {
            "filename": "openwrt.img",
            "version": "15.05",
            "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
            "filesize": 55050240,
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
            "compression": "gzip",
            "compression-target": "openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        }

(where the "openwrt-15.05-x86-kvm_guest-combined-ext4.img" file inside the "openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz" archive will be extracted, and renamed to "openwrt.img")

[boenrobot]

Side note: I would have implemented the above myself, and submit a PR for it, but I'm having a hard time finding the exact point at which GNS3 would automatically download an image that's not already in your "images" folder beforehand. I'm only seeing the code that checks if the image is there, and the config that is executed if the image is already there.

[julien-duponchelle]

It's because we don't download it from GNS3 for the moment.

[julien-duponchelle]

We just open the browser for the moment.

[julien-duponchelle]

About auto decompress.

The problem is we need to support a lot of different algorithmes. For all format and implementation supporting file larger than 1 GB. It's possible but require carefull selections of library or uncompress only some standard format.

Also how do you handle file in subdirectory inside the archive?

[julien-duponchelle]

We should not auto mount it because the md5sum is not correct until you uncompress.

Thanks for the feedbacks !!

[boenrobot]

Odd... I had cleared my images folder, and imported an appliance, without specifically clicking "Download" for its image. Not only was the installation a success, but I checked the images folder, and the image was there... So this made me think that the image is automatically downloaded at some point.

Soooo... we have a feature that's actually a bug?

(This is the 1.4 branch; Local 1.4 branch Windows server with the appliance blocking line commented out)

Also how do you handle file in subdirectory inside the archive?

That's what the proposed "compression-target" is about - a path relative to the root of the archive, so a file in a subfolder inside the archive would be

"compression-target": "subfolder/openwrt-15.05-x86-kvm_guest-combined-ext4.img"

(with the target being extracted in the qemu images folder, with a filename specified in "filename"; All other properties like md5 and size are already - and will continue to - refer to the compressed file. It's assumed that if the archive is fine, so it will be the decompressed one)

Implementation wise, the archive reading library should support listing the subdirectories. The built in "zipfile" and "tarfile" can handle many of the registry formats, though admittedly, not all.

https://github.com/wummel/patool

[boenrobot]

Ooooohhh.... OK, today I also tried to clear my downloads folder, and now indeed, I get "Missing files", and can't install the appliance.

When I download the image into my downloads folder (which is NOT where the .gns3a file is BTW), and refresh, the image is then available, and the source of this effect seems to be this line. The fact that GNS3 scans my downloads folder was what was escaping me.

[boenrobot]

OK, I'm starting to think then that the gns3a file should include both the uncompressed and compressed file size and checksums, so that either one can be used if available. And on closer inspection, the current ones address the uncompressed one, which seems counter intuitive, but it makes sense I guess.

So compressed images IMHO should be reworked so that they're specified together with their uncompressed counterpart, with each being clearly grouped, e.g. in

  {
    "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
    "version": "15.05",
    "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
    "filesize": 55050240,
    "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
    "alternatives": [
      {
        "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
        "md5sum": "6834688e4980f3629fd874700a8fd810",
        "filesize": 4046349,
        "compression": "gzip",
        "compression-target": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
        "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
        "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz"
      }
    ]
  }

the "direct_download_url" and "compression" are moved into "alternatives", since they're referring to the compressed image, not to the uncompressed one. The new "alternatives" would be a collection of files, each of which could be used in place of the original one. They may not necessarily be compressed versions of the original, though appliance makers would be expected (by convention) to usually do just that. Ideally, the file should not be considered missing if one of its alternatives is available instead, and if compressed, should be automatically decompressed on install (I suppose initially, the user could merely be prompted to decompress it? If the gns3a file is made properly, the uncompressed version would be either the main image or an alternative that would then be found, thus keeping UX somewhat smooth, even if a little jarring due to the decompression).

The original can itself be specified as a compressed image, with the "alternative" being the uncompressed.

UI wise, I'm thinking maybe make the Download button open up a menu of possible download locations (spread across the alternatives). Preferably a dropdown accessible from an arrow next to the text. Ideally, pressing that button should not be required, and GNS3 would automatically get the first file with "direct_download_url" (be it the original, or an alternative) and fallback to the original's "download_url", and fail if that download doesn't match, prompting the user to open up their browser to get the file.

Thoughts?

[julien-duponchelle]

Interresting

I propose a small change:

  {
    "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
    "version": "15.05",
    "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
    "filesize": 55050240,
    "locations": [
      {
        "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
        "md5sum": "6834688e4980f3629fd874700a8fd810",
        "filesize": 4046349,
        "compression": "gzip",
        "compression-target": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
        "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
        "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz"
      }
    ]
  }

It's avoid to duplicate the download url and rename alternatives as locations in order to include all locations.

[boenrobot]

But what if the original (uncompressed) itself can be directly downloaded as well? Wouldn't that force a duplication of the first location? Or are download_url and direct_download_url still allowed outside of "locations"? If they are, then... there's hardly any difference with what I'm saying, except the naming (which is fine by me...).

[julien-duponchelle]

It seem more simple to have all download location in array instead of having stuff in two different place

[boenrobot]

So force duplication in that scenario? f.e.

{
    "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
    "version": "15.05",
    "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
    "filesize": 55050240,
    "locations": [
        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
            "filesize": 55050240,
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        },
        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
            "md5sum": "6834688e4980f3629fd874700a8fd810",
            "filesize": 4046349,
            "compression": "gzip",
            "compression-target": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz"
        }
    ]
}

?

Well, fine, I guess that's acceptable, if needlessly verbose.

[julien-duponchelle]

Yeah because download_url could also be different if you have mirror on multiple server.

{
    "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
    "version": "15.05",
    "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
    "filesize": 55050240,
    "locations": [
        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
            "filesize": 55050240,
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        },
        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
            "filesize": 55050240,
            "download_url": "http://mirrors.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://mirrors.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        },
        {
            "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz",
            "md5sum": "6834688e4980f3629fd874700a8fd810",
            "filesize": 4046349,
            "compression": "gzip",
            "compression-target": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
            "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz"
        }
    ]
}

In this case you have a lot duplications.

[julien-duponchelle]

I propose to push this to later release. Perhaps the 2.1 could be focus on image management (add, remove, ...)

[julien-duponchelle]

I push it to the 2.1 (with no warranty) it's too complex to embed in 2.0 we need to move more stuff to server before.

[grossmj]

@noplay is there anything more that needs to be moved to the server?

[julien-duponchelle]

All the appliance management code :( Today the GNS3A import is still made on GUI side. It's a big work to move it.

[grossmj]

Ref https://github.com/GNS3/gns3-gui/issues/2202

[grossmj]

I think https://github.com/GNS3/gns3-registry/issues/506 will help for this issue. In the meantime I think we should introduce compression_target in the appliance version 8: https://github.com/GNS3/gns3-registry/issues/734 to help with future development. We could also support mirrors at at future stage.

This could look like this:

{
    "filename": "openwrt-15.05-x86-kvm_guest-combined-ext4.img",
    "version": "15.05",
    "md5sum": "c8f2635f1cc637b20bbf2fccefb77a1a",
    "filesize": 55050240,
    "download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
    "direct_download_url": "http://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
    "mirrors": [
        {
            "download_url": "http://mirrors.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://mirrors.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        },
        {
            "download_url": "http://mirrors_xxx.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/",
            "direct_download_url": "http://mirrors_xxx.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-kvm_guest-combined-ext4.img"
        },
    ]
}