New appliance format (version 8)

[grossmj]

Discussed in https://github.com/GNS3/gns3-registry/discussions/729

^{Originally posted by **grossmj** January 16, 2023} With GNS3 v3.0 around the corner, it would be a great opportunity to change the appliance file format to add flexibility for custom settings / hardware configuration for specific versions and also add some other improvements: Here is what I have in mind for now: * Support for more than one emulator settings / hardware configurations. One config can be assigned to a specific version using its name. The default config is used (`default` field must be configured if there are multiple configs) or the only config present if none is explicitly assigned to a version. Should we refer to this as "configs" or "settings"? * Support for different checksum algorithms by adding a `checksum_type` field (md5, sha256 etc.) and replacing `md5sum` by a `checksum` field in the `images` section. What algorithms should we support in addition of md5 and sha256? * Should we add `default_username` and `default_password` fields? Example: ``` { "appliance_id": "709c2a9b-5dc3-4362-b147-fb848a0df963", "name": "My appliance", "category": "router", "description": "This is my new appliance", "vendor_name": "Cisco", "vendor_url": "http://www.cisco.com/", "documentation_url": "https://www.cisco.com/c/en/us/support/routing/xxx", "product_name": "Appliance product xxx", "product_url": "https://www.cisco.com/c/en/us/products/xxx/index.html", "registry_version":8, "status": "experimental", "maintainer": "GNS3 Team", "maintainer_email": "developers@gns3.net", "usage": "This is how to use my appliance", "default_username": "cisco", "default_password": "admin", "symbol": "router.svg", "first_port_name": "eth0", "port_name_format": "eth{port1}", "settings": [ { "name": "Default settings", "type": "qemu", "default": true, "adapter_type": "e1000", "adapters": 2, "ram": 4096, "cpus": 1, "hda_disk_interface": "scsi", "arch": "x86_64", "console_type": "vnc", "boot_priority": "cd", "kvm": "require", "options": "" }, { "name": "Custom settings for version 7.10.2", "type": "qemu", "adapters": 4, "ram": 8192, "cpus": 1 } ], "images": [ { "filename": "file.iso", "version": "7.10.2", "checksum": "ef8712e655fcbc92dc1a1551ee2e4a80", "checksum_type": "md5", "filesize": 1287245824, "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/7.10.2" }, { "filename": "file2.iso", "version": "6.10.4", "checksum": "e5b844cc57f57094ea4585e235f36c78c1cd222262bb89d53c94dcb4d6b3e55d", "checksum_type": "sha256", "filesize": 3949459594, "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/6.10.4" }, { "filename": "empty100G.qcow2", "version": "1.0", "checksum": "1e6409a4523ada212dea2ebc50e50a65", "checksum_type": "md5", "filesize": 198656, "download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/", "direct_download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/empty100G.qcow2/download" } ], "versions": [ { "name": "7.10.2", "settings": "Custom settings for version 7.10.2", "images": { "hda_disk_image": "empty100G.qcow2", "cdrom_image": "file.iso" } }, { "name": "6.10.4", "images": { "hda_disk_image": "empty100G.qcow2", "cdrom_image": "file2.iso" } } ] } ``` Please let me know if you have comments / suggestions.

[eantowne]

Should we refer to this as "configs" or "settings"?

• I think "settings" may be better so as to not cause confusion with the configuration within the given OS. What algorithms should we support in addition of md5 and sha256?
• Honestly I think that, from a security perspective to prevent/reduce the chance of a malicious actor taking advantage of a collision, SHA256 would be the minimum. Most vendors are providing SHA256 and it seems to be the general standard. I truly do not see a reason for providing multiples, beyond recursive validation by a user for manual download to compare the hash in the .gns3a/Marketplace vs the vendor provided hash. Should we add default_username and default_password fields?
• I think this is a glorious idea!

[grossmj]

Thanks @eantowne, I will also use your ideas from the discussion to create the new format. Also, I think we should just make it a evolution from the current format (aka version 8) and still support older versions in GNS3 v3.0 otherwise it would be a nightmare to maintain two different branches for appliances.

[eantowne]

I would agree with enabling backwards compatibility. I would use version 30 though, gives plenty of room for updates to the 2.x branch, and clearly delineates itself as the new and improved contestant.

One thing does occur to me though, and that is, how is this going to work for the Marketplace? For instance, my intention is to create a whole bunch of updates for a wide variety of appliances to take advantage of the new file capabilities. This will basically create duplicate .gns3a's for certain appliances. Does there need to be some form of check by the Marketplace sync to look if the .gns3a is 3.0 compatible and then label it as such? Or would it be easier to have two separate appliance list in the Marketplace? Even then, a 3.0 controller will be able to read and import a v4 or 3.0 compatible, but then when do we stop updating 2.x compatible appliances?

[grossmj]

One thing does occur to me though, and that is, how is this going to work for the Marketplace? For instance, my intention is to create a whole bunch of updates for a wide variety of appliances to take advantage of the new file capabilities. This will basically create duplicate .gns3a's for certain appliances. Does there need to be some form of check by the Marketplace sync to look if the .gns3a is 3.0 compatible and then label it as such? Or would it be easier to have two separate appliance list in the Marketplace? Even then, a 3.0 controller will be able to read and import a v4 or 3.0 compatible, but then when do we stop updating 2.x compatible appliances?

Actually I was thinking we would update appliances to take advantage of the new file capabilities only when needed, for instance if we need different hardware configurations. I know this is not ideal and that would mean these appliances cannot be used in GNS3 v2.x

I think there are 2 possible solutions:

• Copy all .gns3a appliances to a new directory in the repo and convert them to the new format. The marketplace would then have two separate appliance lists with a toggle button to switch between v2 and v3. Main problem is we would have to maintain 2 separate lists for a while until v3 is well established (this could be for a long time, especially I predict some users will want to stay on v2 because the local server is supported on Windows).

OR

• We keep the current .gns3a files, upgrade the ones that should use the new capabilities and support the new format in GNS3 >= v2.2.38. This would involve some major work from us and and also force everyone who want to use appliances with the new format to upgrade their GNS3 version.

[grossmj]

Copy all .gns3a appliances to a new directory in the repo and convert them to the new format. The marketplace would then have two separate appliance lists with a toggle button to switch between v2 and v3. Main problem is we would have to maintain 2 separate lists for a while until v3 is well established (this could be for a long time, especially I predict some users will want to stay on v2 because the local server is supported on Windows).

We could also have 2 different branches, master for v2 and 3.0 for v3. Then we would just merge changes done on master into 3.0

[eantowne]

We could also have 2 different branches, master for v2 and 3.0 for v3. Then we would just merge changes done on master into 3.0

Wouldn't this merge be a massive issue?

For instance Juniper vMX current .gns3a does not support the ability to have a multi-RE/PFE configuration. The new format would. How does the 2.x format merge into the 3.0 format without causing issues, or at least confusion if going with a 2 file model.

I think there may be an argument for just keeping the 2.x and 3.x branches completely separate. If the 2.x branch does not get "official" updates, but the 3.x does, this might also help push people to upgrade to 3.x if they cannot find the template they need in 2.x, but it is sitting there in 3.x.

There comes a point where supporting the "old" versions may not be worth the effort. I am not saying just immediately stop all 2.x registry work the day that 3.x is released, but I think a planned obsolesce might be a good thing. Announcing an end-date for 2.x registry support projected out X months from 3.x release?

[grossmj]

Wouldn't this merge be a massive issue?

Correct, this is why will we introduce a version 8 of the current format and support it in both >= v2.2.41 and 3.0. This means a bit more work for us but I believe this will create less headaches down the road.

[grossmj]

I have tried to include all the suggestions but please let me know if you see something that needs to be added/changed. Support for more that 4 drives for Qemu VMs will be added later in 3.x so that should be added in another version of the appliance format.

Summary of what this version will support:

• uefi boot mode option for Qemu
• Possibility to have multiple set of settings to use with different image versions. The default settings are specified if 1. a default field set to true exists or 2. there is only one set present. A template_type field must be added to tell GNS3 what template to create (qemu, iou, dynamips or docker), this could allow one appliance to be used to create different template types. All template specific properties are defined in a template_properties field.
• The default settings are inherited by other settings set, this can be blocked by setting inherit_default_properties to false
• The controller template default is used if a template property is not defined or inherited from the default settings.
• New optional emulator_versions field in the settings to restrict using the emulator properties on specific emulator versions.
• The md5sum field is renamed checksum. The md5sum field is still accepted for easier migration from previous format versions.
• New optional checksum_type field for future development. The default and only checksum type remains MD5 for now.
• New optional default_username and default_password fields at the appliance and version levels.
• New optional installation_instructions field at the appliance and version levels to give download/unpack instructions to some appliances. See https://github.com/GNS3/gns3-registry/issues/506
• New optional compression_target field to be used along with the compression field in future development.
• The idlepc field in versions section is moved to the template_properties for the dynamips template type
• The first_port_name, port_name_format, port_segment_size and linked_clone fields are moved to the template_properties for the qemu template type (these fields are only valid for Qemu templates).
• The arch field for qemu has been renamed platform to match the template properties on the controller side.
• The kvm field has been dropped and no longer required. Installing an appliance shouldn't take into account the available servers and their capabilities (e.g capable of running kvm etc.)
• The category, usage and symbol fields can be defined in any template_properties. Defaults are at the appliance level or version level (they will be injected in template_properties if there aren't already defined there).
• The version installed will be injected in the version field of the template (controller version >= v3.0)
• Add name and default_name_format fields to all template properties
• Add console_resolution, extra_hosts and extra_volumes to Docker properties
• Add "spice+agent" for console_type in Qemu properties

Example:

{
    "appliance_id": "709c2a9b-5dc3-4362-b147-fb848a0df963",
    "name": "My appliance",
    "category": "router",
    "description": "This is my new appliance",
    "vendor_name": "Cisco",
    "vendor_url": "http://www.cisco.com/",
    "documentation_url": "https://www.cisco.com/c/en/us/support/routing/xxx",
    "product_name": "Appliance product xxx",
    "product_url": "https://www.cisco.com/c/en/us/products/xxx/index.html",
    "registry_version":8,
    "status": "experimental",
    "maintainer": "GNS3 Team",
    "maintainer_email": "developers@gns3.net",
    "installation_instructions": "This is how to install this appliance",
    "usage": "This is how to use my appliance",
    "symbol": "router.svg",
    "default_username": "cisco",
    "default_password": "admin",
    "settings": [
        {
            "name": "Default template settings",
            "default": true,
            "template_type": "qemu",
            "template_properties":
                {
                    "symbol": "multilayer_router.svg",
                    "first_port_name": "ethernet0",
                    "port_name_format": "ethernet{port1}",
                    "adapter_type": "e1000",
                    "adapters": 2,
                    "ram": 4096,
                    "cpus": 1,
                    "hda_disk_interface": "scsi",
                    "platform": "x86_64",
                    "console_type": "vnc",
                    "boot_priority": "cd",
                    "options": ""
                }
        },
        {
            "name": "Custom settings for version 7.10.2",
            "template_type": "qemu",
            "inherit_default_properties": false,
            "template_properties":
                {
                    "adapters": 4,
                    "ram": 8192,
                    "cpus": 1
                }
        }
    ],
    "images": [
        {
            "filename": "file.iso",
            "version": "7.10.2",
            "checksum": "ef8712e655fcbc92dc1a1551ee2e4a80",
            "checksum_type": "md5",
            "filesize": 1287245824,
            "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/7.10.2"
        },
        {
            "filename": "file2.iso",
            "version": "6.10.4",
            "checksum": "68232f77da8f78cdc9aa6f3266a4d4c0",
            "filesize": 3949459594,
            "download_url": "https://software.cisco.com/download/home/286307342/type/286307754/release/6.10.4"
        },
        {
            "filename": "empty100G.qcow2",
            "version": "1.0",
            "md5sum": "1e6409a4523ada212dea2ebc50e50a65",
            "filesize": 198656,
            "download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/",
            "direct_download_url": "https://sourceforge.net/projects/gns-3/files/Empty%20Qemu%20disk/empty100G.qcow2/download"
        }
    ],
    "versions": [
        {
            "name": "7.10.2",
            "settings": "Custom settings for version 7.10.2",
            "images": {
                "hda_disk_image": "empty100G.qcow2",
                "cdrom_image": "file.iso"
            }
        },
        {
            "name": "6.10.4",
            "default_username": "admin123",
            "default_password": "admin123",
            "installation_instructions": "This is how to install this version",
            "usage": "This is how to use this version",
            "symbol": "ethernet_switch.svg",
            "images": {
                "hda_disk_image": "empty100G.qcow2",
                "cdrom_image": "file2.iso"
            }
        }
    ]
}