Catapult takes over half an hour to back up larger save files

[Terrorforge]

I currently have two worlds, one of which is reset and empty, and one of which is actively played. The active world is 1.3GB.

Creating a manual backup of these worlds took about 33 minutes

I understand that there must be some allowance for larger files being slower, but it does not take 30+ minutes to copy a 1.3GB folder. Just doing it manually through the file explorer takes me less than 10 minutes, creating a compressed archive with winrar only takes 2. 1.3GB isn't even that large as CDDA saves go, and being this slow renders the backup utility borderline unusuable.

[qrrk]

To be honest, even the 10 minutes it takes in the Explorer seem way too long. It this on an HDD?

[Terrorforge]

Oh, yeah. It's sitting on a 10-year-old Caviar Green HDD. I completely forgot that might be an issue because I rarely have problems with it except when running new games that are really heavy on the loading.

My file is only 450mb at the moment, but after transferring to my SSD it backs up in 4 minutes, which is probably fine:

[KawaiiSelbst]

I would recommend moving the backup system to git rather than completely copying the folder, because all the save data is in text format, so it will be very efficient to store it, a world backup of any size won't take more than 30 seconds, saves will be able to be stored on github, you won't have to worry about deleting old backups, because the weight of the repository won't exceed the weight of the save, no matter how many backups it stores in itself.

[ctrenthem159]

Backing up to git would require a significant amount of setup for users, and relies on a git provider to be available. It would also require a lot of handling on the backend to resolve merge conflicts without demanding a user's intervention, which can be a huge problem for people who aren't used to using git normally.

I think the problem at issue is more the person's hardware, not catapult itself. @Terrorforge when you compared the times on file transactions, did you compare using the same folders? Comparing an action using Windows Explorer that only moves a dozen or so large files to an action that is moving thousands of small files is not a valid comparison, even Explorer will bog down significantly when moving many small files on an old or low-quality HDD. It also can be impacted by other processes running at the same time, or if an AV program is scanning everything as it is being moved. There's a lot of other potential causes based on the information provided that aren't catapult's fault. Not saying there isn't room for improvement on catapult's end, just that it might not make as much of an impact as you'd like.

[KawaiiSelbst]

@ctrenthem159

Backing up to git would require a significant amount of setup for users, and relies on a git provider to be available. It would also require a lot of handling on the backend to resolve merge conflicts without demanding a user's intervention, which can be a huge problem for people who aren't used to using git normally.

I think the problem at issue is more the person's hardware, not catapult itself. @Terrorforge when you compared the times on file transactions, did you compare using the same folders? Comparing an action using Windows Explorer that only moves a dozen or so large files to an action that is moving thousands of small files is not a valid comparison, even Explorer will bog down significantly when moving many small files on an old or low-quality HDD. It also can be impacted by other processes running at the same time, or if an AV program is scanning everything as it is being moved. There's a lot of other potential causes based on the information provided that aren't catapult's fault. Not saying there isn't room for improvement on catapult's end, just that it might not make as much of an impact as you'd like.

You are not quite right, to interact with git it is not necessary to have an official git desktop client, besides in this case you do not need all git functions only commit and checkout, github is full of implementations of basic git functions completely in code, besides conflicts are formed only in cases when in the same file after one and the same commit 2 different changes are made in the same place, and somehow there are 2 different commits that you want to merge, mostly this happens when several authors are working on a repository. And the benefit of this approach is obvious from the first glance specifically in the disk space occupied, it will be negligible, and as a nice bonus the increase of backup speed, and multiple backups, without additional disk space requirements.