search

fmd-project-team / FMD

The new FMD fork! Join our community on Discord!
https://discord.gg/cXKKgw3
GNU General Public License v2.0
263 stars 33 forks source link

Corrupted archives sometimes / Validate archives after download #8

Open mastafrooper opened 5 years ago

mastafrooper commented 5 years ago

It's not super regular but occasionally, normally when downloading a large series, some cbz files are corrupt.

Was wondering if at some point in the future an option could be added to validate the zip/cbz when it is finished and if corrupted redownload/recompress before moving on.

A temp directory for chapter packing.

SDXC commented 5 years ago

Do you mean the file packing mechanism of FMD itself or do you download cbz from the website? Is the archive itself corrupt or just some images?

This corruption should be fixable if it's being zipped after the download.

mastafrooper commented 5 years ago

The packaging mechanism of FMD itself.

The archive itself is corrupt. If you run a test on it with 7zip the error is: 'Unexpected End of Archive' and it won't open with any readers. Tell it to download the chapters again and it works fine.

FMD is running in a VM saving to a network location. That location is on the host machine though so there should be no network drops.

SDXC commented 5 years ago

Okay, thanks for the reply. That's a bug then.

It either could be fixed with a new version of 7za (if it is the problem) or by fixing the way FMD writes the files to the archive or moves the archive (if it does). It can actually be a problem on the networking side, but more on the application layer than the actual network layer.

SDXC commented 5 years ago

I'll take validation for archives in as a feature request. It would be a new option next to the archiving option.

mastafrooper commented 5 years ago

Thanks. No hurry at all, it's been happening for a long time I've just been too lazy to log an issue. The move to a new fork kinda just motivated me some ;D

SDXC commented 5 years ago

I have updated 7-Zip from 18.01 to 19.00. There were many bugfixes between them, so it's possible that it already fixes your problem. The new version of FMD will go public in the next hours. After the update please observe the behavior and let me know after some time, if it got any better, still the same or even worse.

On a side note: the new version of 7-Zip should be a little faster.

PS: Ticket won't be closed if this fixes your problem, as the validation part is still a good idea. It just won't be high priority.

mastafrooper commented 5 years ago

I'll give it a try when I get home from work... in like 8 hours.

The problem isn't super consistent so I'll give it a few days and maybe find a new series to download at the same time.

SDXC commented 5 years ago

Official release is out now. The updater in the version 0.9.160.0 doesn't work. You have to update manually. Updater should work for the future releases though.

mastafrooper commented 5 years ago

Problem is still occurring. It's random so I can't be certain but the only thing I can think of is when it is downloading when my Surface Go is turned on as it's set to sync my manga folders whenever there are changes.

It shouldn't be locking a temp file though unless FMD isn't locking it. It doesn't happen to every file though. This series I found had 14 chapters and it happened to 4. I also downloaded another 9 chapters of other stuff before that and all those files were fine.

20:02:37:568 EFCreateError - DoZipCbz.Exception Unable to create file "\\ABYSS-SVR\Data\Manga\To-Read\Maou-Sama Retry\0004 - Vol. 01 Ch. 004.tmp" (15 Items) $000000010004BAFB $000000010004B9F3 $00000001004A9AB2 $00000001004A9E10 $00000001004A9CF2 $00000001004A9C44 $00000001004A5D08 $00000001004A63D7 $00000001002767A1 $0000000100278CF5 $0000000100053E9D $0000000100003B76 $0000000100015047 $00007FFE2B7E81F4 $00007FFE2E26A271

SDXC commented 5 years ago

Can you recreate this issue with the exact same series over and over again? If we can recreate this more or less at will, we can test better.

Regarding the problem itself, here are some guesses:

Things to test:

Things to check on your side:

Things to check on my side:

mastafrooper commented 5 years ago

Did some more testing last night.

Does seem to be the syncing that's causing the issue. Added exclusions to it for JPG, GIF, PNG and a few other image formats I could think of as well as TMP files. Also tweaked it's idle time to 60 seconds after last modification before it syncs. Still got the occasional corrupt CBZ. Shut the tablet down when I went to sleep and another 100-150 chapters of things downloaded without incident.

So it seems like the sync tool is whats causing FMD to not archive things correctly. Whether it is ignoring exclusions or it is locking something when it is checking timestamps etc I have no idea.

I suppose a file lock/temp file directory might sort the issue. Hardly a super important issue though since it is caused by my weird setup.

SDXC commented 5 years ago

So it's not an actual bug then. I will implement an option to define a temp directory for chapter packing. Meaning that chapters will be downloaded to temp, zipped, archive moved to actual target and temp will be cleared afterwards.

I will evaluate some more concepts though.

The file locking thing is something that 7-Zip has to do, but you can't instruct 7za to do that. On the other hand 7-Zip actually locks the file, but the sync could be a bit too aggressive.

mastafrooper commented 5 years ago

Awesome. No hurry though as this is something normal people wont deal with ;D