Export portable project fails with included large base images

[ahorn42]

Hi,

first of all, thanks for that great piece of software!

I discovered an issue with the export functionality. If I am exporting projects (from GNS3 UI or the web interface) with out the base images included till now everything worked nice. But if I enable the export of base images, this seems to fail, if the images reach a certain size. For reference, a ~8GB image of an Ubuntu Desktop worked, whereas a ~22GB Windows 11 image fails with a timeout error. Both are vmdk images.

The error message I get is:

ExportProjectWorker thread stopping with an error: Timeout when downloading image 'WinDev2212Eval-disk1.vmdk' from remote compute

(also there seams to be a typo, as the r in computer is missing)

I am using GNS3 UI on windows, with a GNS3 VM in Hyper-V, the VM has 16GB of RAM and 8 vCPU cores, the download speed from the VM is about 300Mbit/s (in case that matters).

After a bit of research I discovered, that there have been issues with timeouts in earlier versions, but the code should have been changed to an implementation without timeout, so I am a bit confused how/why this error occurs. (see: https://github.com/GNS3/gns3-server/issues/1338)

Can anyone confirm this or has an work around?

Can I for example simply export the project without base images and then modify the exported file and include the images in an images folder? Would this be sufficient or do I need to change something else?

I would really like to avoid to copy the images manually from server to server, as this might easily lead to inconsistencies in how we want to use it.

In addition
I am using GNS3 2.2.44.1

[ahorn42]

Also i noticed, when downloading the images via scp I reach a download speed of 800-900Mbit/s compared to 300 during the export - but this might also be related to the compression capabilities (I am not sure how exactly the export process works).

[josephmhiggins]

i am only using gns3 <= 2.2.43 When you export a project, you can do anything you want to the exported project because it is a backup. All my projects that I have previously exported on Windows 10 were set to have the nodes in the project file as: "compute_id": "local", I used 7-Zip to open the archive and then open the .gns3 file and change it to: "compute_id": "vm",

With powershell you can definitely add files to the archive (it's like 4 lines of code) - super simple. With 7-Zip, I am extremely confident you can do it with the GUI, but I have never done that. 7-Zip is really nice.

"Also i noticed, when downloading the images via scp I reach a download speed of 800-900Mbit/s compared to 300 during the export" - yeah, well, the gns3 way probably works for all operating systems and is all kinds of backwards and forwards compatible.

That being said - 800-900Mb/s seems like an extremely high number. I got 40vcps - with 256G of RAM - i will have to test it on esxi and compare.

[josephmhiggins]

if you can not upload the project to the gns3 marketplace (22Gig is crazy) or free google drive account, then strictly speaking I view your design as illegal. I would just left click on the GNS3 VM and copy and paste the whole thing to an archive and download a new GNS3 VM. The GNS3 VM is about 1G (not sure how big it is) and people have projects that use 256G. Left-click copy paste and let windows or your operating system figure it out.

[josephmhiggins]

With 40 vcpu, I can export a project, e.g. 20G, and copy and paste the entire gns3 vm at the same time. 8vcpu is a little light. My technique of doing that (I do not think I will ever create a gns3 project with 20G in it) affords me 2 different logical backup paths. 1 path goes through my operating system. 1 path goes through the gns3 developers. The only thing I have to worry about is my CPU Temperature. I just spent 48 hours backing up 7 years of GNS3 so... For a measly 20g project, I would not buy 10G NICs. For a 256g project, I would definitely buy 10G NICs.

For the "timeout errors"... yeah, I discussed it 2.5 years ago - best guess. I am very, very, very sorry, but a 20G project is not a valid project design - unless you are going to simply copy and paste the entire gns3 vm - prefereably over a 10G link.

[josephmhiggins]

the best ancillary support to gns3 was wireshark, until the best ancillary support to gns3 became export/import project gns3 export/import project is 100 times more powerful than wireshark for long time gns3 users the gns3 rest api, i am sorry, it is kinda complicated so, imho, any complaints about export/import project are valid because it is very, very powerful

[ahorn42]

Sorry - I am a bit confused about your comments - maybe there were some misunderstandings.

Regarding the projects size - the 22GB are not the size of my GNS3 project, it is the size of the base image of a Windows 11 VM I am using within my GNS project (and for a full Windows installation, 22GB are quite small ;) ) - the project without base images is just a few hundreds MB.

Regarding the download speed of 800-900Mbit/s - that was the speed I was able to archive when download large files (the 22GB base image) from the GNS3 VM to my Host machine (Windows 10) via scp (of the WSL2 inside my Windows 10 Host), so the connection was between my Host and the GNS3 VM running at that same host. It was neither a connection speed withing a GNS project nor between different physical machines. The host is a well equipped desktop PC from 2019, but nothing too special (i9 9th generation, 128GB of RAM and SSD storage).

Furthermore I don't really get what you want to tell me about wireshark and how it is connected to my problem?

I simply want to export my GNS project including the base images used by this project via the GNS UI, so that I can take the *.gns3project file and take it to another PC without worrying about keeping the base images in sync, as the 2nd PC is not connected to the internet. And during that export process I experienced the timeout error I reported above.

[josephmhiggins]

the timeout problem has existed for large size project (i.e. with vm base images included) for, approximately, 2.5 years. Someone reported it here. I looked at here it. I confirmed it here. I came to the conclusion that it was unwise to include base images in an exported project. If your off-grid pc has the same operating system, then copying and pasting the entire gns3 vm will do the trick.

[ahorn42]

Allright, thanks for clarification :)

Well in my case it is an migration from Windows Host with VM to a Linux Host without VM but I got it working by manually packaging the images into the project file. And then I discovered that it also seems to be enough to simply copy the images in the images folder of the GNS project folder under <home>\GNS3\images. I thought importing images would be more complicated ;)

Is there any help I might can offer to fix this timeout issue? Sounds like it should be fixable. On the other hand I understand, if large base images are not the intended use case. But then maybe a hint near the export function or in the error message would be nice.

[josephmhiggins]

"But then maybe a hint near the export function or in the error message would be nice." - ummm, yeah. I would probably ....if i was a developer.....block an export project greater than........250MB.

The gns3 export project was declared by the developers to be completely redesigned in gns3 version 3.1. We are still at least weeks away from gns3 version 3.0 I was, until recently, a 7 year user of VMware and Windows 10. I would have to look at the VMware logs and the GNS3 VM syslog. The VMware logs are nothing like the gns3_server logs. The gns3_server logs are wonderful. The VMware logs are horrible. I suspect VMware, and/or, Ubuntu are moving files around (defragmenting the hard drive) when the system is dormant - an export project only reads from the "hard disk". ......so it is best to leave this to the developers and issue a workaround.

[josephmhiggins]

hrmmph, nothing has changed in programming in the last 30 years. lol, i suspect they got a null pointer to a file that no longer exists at the specified location. i would make it: exportProject { copyProject() removeProject() } that would keep ubuntu and vmware busy with me writing and deleting stuff

[grossmj]

Exporting has been greatly improved in version 3.0 (currently in beta), it uses zstandard compression which is much faster.