Add a second progress bar for the saving portion when Exporting

[SteveHutchinsASBFA]

Hello,

The progress bar in the task window (under "State") when exporting books seems to reach 100% before compressing the new cb* file. So, especially for large books, the progress bar can sit at 100% for quite some time, giving no indication how far along the process truly is. Just a nuisance, but thank you for taking a look.

--Steve

[maforget]

As a test I've converted a 4GB (biggest I got) book to the new AVIF format. It stayed at 100% for a second no more.

What I believe you are seeing is that the actual progress bar seems to count the pages when they are being converted. After that it writes the pages to disk. I believe it is this part you are seeing that stays at 100% longer.

Please see this Wiki entry where I did a benchmark, after increasing the number of thread for converting. The time is split in Converting and Saving. You can see that the Saving part can take from 1.5 to 10 sec depending on the file size. When converting to another type like AVIF or WEBP, the saving portion is inconsequential in regard to the total time.

In my example at the beginning that file that seem to take a second went from 4GB to 250MB. But converting the original file stayed at 100% for at least a minute, because of it's enormous size and the drive taking awhile to write it. It might depend on a panoply of factors, like drive speed, the settings you did, was a temp file used, etc.

Since it is iterating thought the files a second time, just halving the percentage would be widely inaccurate because the converting that the bar tracks usually takes the longest. Only thing I can see is having 2 bars one for converting and another saving. Just not certain it is a problem that will be encountered frequently enough.

[maforget]

I've done some testing and I don't think there is a good answer here. I've tried changing the progress bar so that the converting part will be max 90% and the remaining 10% is the saving portion.

It works half the time. If the target file doesn't exists, it works correctly. If it already exists and you are only converting an existing file, it creates the file in a temp folder and moves it to the target location. The move part is what seems to take awhile. When your system drive is different from your target drive, it will have to move the file across drives and that is the part that is slow. Especially when copying to a slow HDD.

Also the part of the code from the framework that handles this doesn't report progress. I would have to implement a custom move function that reports the progress and honestly at this point it is already a lot of changes. For something that will happen rarely. Only when a big file is created, across multiple drives, just so that the bar doesn't hang for a couple seconds. Not even sure if it's worth it pushing the test I did with splitting the bar progress to 90% / 10%.

[SteveHutchinsASBFA]

Thank you for looking into this; I appreciate the time you spent.

It's interesting. I'm converting compressed files (of various types) containing usually JPEGs to CB7 files containing WEBP. The files are moving from one external hard drive to another.

I tried converting and re-compressing a file without moving it and the time was roughly the same. However, that was still on an external hard drive, so maybe temp files are being created on the Windows drive and it's still really a move?

The file was ~100MB and took about five minutes in total.

So, yes, I think you're right and it's a combination of settings and destinations, etc.

No worries if you don't think it's worth it. I agree. I really only have experience coding in VBA, but I do remember trying to come up with progress bars there and it was just impossible to reflect the real world.

Thanks again for your time!

On 08/09/2024 10:00 PM EDT maforget @.***> wrote:

I've done some testing and I don't think there is a good answer here. I've tried changing the progress bar so that the converting part will be max 90% and the remaining 10% is the saving portion.

It works half the time. If the target file doesn't exists, it works correctly. If it already exists and you are only converting an existing file, it creates the file in a temp folder and moves it to the target location. The move part is what seems to take awhile. When your system drive is different from your target drive, it will have to move the file across drives and that is the part that is slow. Especially when copying to a slow HDD.

Also the part of the code from the framework that handles this doesn't report progress. I would have to implement a custom move function that reports the progress and honestly at this point it is already a lot of changes. For something that will happen rarely. Only when a big file is created, across multiple drives, just so that the bar doesn't hang for a couple seconds. Not even sure if it's worth it pushing the test I did with splitting the bar progress to 90% / 10%.

— Reply to this email directly, view it on GitHub https://github.com/maforget/ComicRackCE/issues/98#issuecomment-2278937564, or unsubscribe https://github.com/notifications/unsubscribe-auth/BKL63DUIDI25TBH5F3QH2OLZQVX43AVCNFSM6AAAAABMIPS556VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZYHEZTONJWGQ. You are receiving this because you authored the thread.Message ID: @.***>

[maforget]

maybe temp files are being created on the Windows drive and it's still really a move?

Yes it uses your computer temp file which is usually on the windows drive. You won't really see a change whither or not you replace a file. It takes around the same time to copy the file than to write it directly, especially on slow external drives. The change was visible when I added the writing part to the progress bar. It would still hang when replacing because it adds a copy step.

[SteveHutchinsASBFA]

I just did a comparison between external hard drive and internal hard drive. Working from the internal hard drive to the internal hard drive was faster, and I mean RIDICULOUSLY faster.

So, I see what you're seeing now with the difference being a matter of seconds.

On 08/09/2024 11:50 PM EDT maforget @.***> wrote:

maybe temp files are being created on the Windows drive and it's still really a move?

Yes it uses your computer temp file which is usually on the windows drive. You won't really see a change whither or not you replace a file. It takes around the same time to copy the file than to write it directly, especially on slow external drives. The change was visible when I added the writing part to the progress bar. It would still hang when replacing because it adds a copy step.

— Reply to this email directly, view it on GitHub https://github.com/maforget/ComicRackCE/issues/98#issuecomment-2278965507, or unsubscribe https://github.com/notifications/unsubscribe-auth/BKL63DX7J62MK5DWJ6ZPWXLZQWEYNAVCNFSM6AAAAABMIPS556VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZYHE3DKNJQG4. You are receiving this because you authored the thread.Message ID: @.***>