Files greater than 2 GB can't be converted

[5h4k3]

It seems that files larger than 2 GB cannot be converted with ACBT or ACBR. And why are they so big? Lossless PNG to work on some Omnibus or Compendium who are about 1000+ pages.... I have never seen that before because I never try to convert more great than a TPB until now. But this one in the screenshots is about 2.5 GB and the greater I faced, was around 6 GB... It's not very important in case it can't be fixed, i just have to unzip them and voilà. It's more a report for your information if this is not a intended behavior...

Have a nice day !

[binarynonsense]

Never tried a file so big so never encountered this but, from a first quick search, seems to be a limitation of at least one of the libraries I use (adm-zip), the file is uncompressed in memory and there's a hard limit for the buffer size of 2 GB...

If you have a file of that size in one of the other supported formats (especially cbr, cb7) and can test if it also fails let me know.

Can't look more into it right now, don't now when I'll be able to work on the tools or ACBR, but if I have time in the future I'll see if I can find a workaround at least for ACBR at first (if zips are the only ones having this issue I probably could switch, or have an option to, to uncompressing them using 7z, which I already use for other things and may not have this problem... I'd need to make some tests).

Thanks for the bug report!

[5h4k3]

I had thought of something like a limitation...

But I hadn't thought trying with a 7z / cb7 or rar / cbr... And, you were right, it's seems to be Zip limited. Or at least, it works with 7zip ones...

I haven't test with Rar because I don't have one under the hand and I can't compress in Rar (don't have Winrar but 7zip).

So, seems a nice workaround for now and like I said, it was not very important...

I will let you close the issue or keep it it open if you prefer..? Thanks for the fast response ! And good luck in your other projects !

[binarynonsense]

Let's keep it open in case someone has the same problem / question.

No problem, thank you for letting me know. If I can work on it and manage to fix it some day (I'll have to test it and think about the best solution but seeing that the problem doesn't seem to happen with 7z that will probably let me work around the issue, as it also can be used to extract zips) I'll let you know here and close it then.

Thanks!

[tactiical]

I'm running into the same issue with both .cbz and .cbr large files, however .cb7 files seem to work. Only on ACBR though, ACBT errors out.

*edit formatting/rewording

[binarynonsense]

Thanks for the feedback!

This matches my conclusions when I looked a little into this some time ago, right now cbr and cbz files greater than 2GB can't be converted (edit: or read, if I'm not mistaken) using ACBR, or ACBT (didn't check cb7 files using ACBT but I'm not sure right now why they would fail with it and not ACBR), and the reason seems to be that the libraries I use for extracting/compressing those formats can't work with files that big (if I remember correctly, they need to load the file into memory and the biggest buffer they can create is 2GB).

I have some ideas I want to explore to see if I can add alternative ways to make things work for such big files (the library I use for cb7 files can also work with cbz so that should be a good alternative, for cbr I'm still not sure but have some things I want to try) but I haven't had the time yet to work on it so, for now, it's sadly a limitation of ACBR and ACBT.

[tactiical]

No worries! Resizing and converting my 5.6GB .cb7 file worked through ACBR. I don't mind having to convert them to .cb7 first before resizing. Just glad one format works :) Thanks for all your work.

[binarynonsense]

Glad you found a way to do it.

Was that the same cb7 file that didn't work with ACBT? And is the resulting file bigger or smaller than 2GB?

I'm a bit confused as to why converting the cb7 to cbz works on one and not the other if you are trying to do the same operations and the same input and output formats. (edit: Although ACBR and ACBT internals are a bit different,) Extracting the cb7 should always work as the library for cb7 doesn't seem to have the size limit, but both programs should have the same problem when it comes to compressing the output cbz if there's one... I'm not completely sure if the library has the same limitation when it comes to compressing files if the output file is going to end up being greater than 2GB (maybe not, haven't tested it) but if it does, both programs should fail and if not, both should succeed... maybe I do something different in ACBT I don't remember :) (ACBT is a bit of an afterthought and haven't updated it much, so I may have made changes in ACBR later on that cause the discrepancy... but can't think of a reason for the difference right now)

[tactiical]

Yeah, it was the same cb7 that failed in ACBT. I tried them both at the same time. ACBT failed and spit out the error in the screenshot from my previous post. ACBR completed the entire process. Although I didn't do any resizing when I attempted with ACBT, just converting to cbz.

And yep, the resulting file was properly compressed. I set it to 50% resize, 80 img quality in ACBR and it came out at 1.44GB

[binarynonsense]

Thanks for the extra info! I'd need to do some tests to verify it but I think I know what's happening and, if so, both probably have the same size limitation when it comes to extract OR create cbz (and cbr) files.

My guess is ACBT failed because the resulting file was bigger than 2GB and ACBR didn't because, as you were also resizing the pages before the final file was created, the output was smaller than those 2GB. ACBR would probably fail if you just tried to convert it from cb7 to cbz.

[tactiical]

Yeah I'll probably keep testing different .cb7 file sizes. ACBR just failed to convert a 12 GB cb7 so it seems like there is a maximum file size.

[binarynonsense]

That's quite a big file :) Even if you managed to reduce the size of the pages by half during the conversion it will still have to output a 6GB cbz file. That seems to confirm my theory that creating zip files bigger than 2GB with the library I'm using by default can't also currently be done (same probably with cbr).

cb7 I'd think would not have that limitation as it doesn't seem to have any problem extracting big files so far (the conversion fails during the creation of the output file), but I'd need to test it.

One test you may try if you are curious and have the time and patience (I'd image the conversion for a file that big will take quite some time) would be doing that same conversion that just failed (big cb7 file to cbz) BUT setting a password in the advanced options, as in that particular case I'm pretty sure I create the cbz output file using the same library I use for cb7, not the default one because it can't create files with a password and that one can (it won't be a practical solution right now for you, as you probably don't usually want the comic files you create to have a password, but this will confirm that that library works for extracting big cb7 files and creating big cbz ones and adding an option to switch to that library for 'normal' cbz creation, and hopefully extraction but that would require other tests, would be interesting for a future version of ACBR).

[tactiical]

Yeah its huge haha, been creating my own Omnibus collections of my comic series and this one has quite a lot.

And I will give the password thing a shot and let you know what happens. For now, I decided to just split the file into parts and then resize the smaller parts by themselves, then remerge them all together.

[tactiical]

Alright I tried it with a password and it error'd out with this message

As a side note, I managed to get the 12GB file down to 3GB by splitting, resizing the parts, and remerging them all. So really not the end of the world if ACBR can't handle massive files.

[binarynonsense]

Interesting... this feels like a different kind of problem. Given the error code (spawn ENAMETOOLONG) my first theory is that your file has so many images that the command I pass to 7z (which includes the paths to all the image files) is too long, so it failed even before really trying to create/compress the file. Maybe I can create a text file with all the file paths and pass that to 7z if it allows that as an input or create the zip file incrementally in more than one command when there are so many files... I'd have to look into it as I've never encountered this kind of problem (none of my test files are this big, not even close, so it was never an issue :))

[tactiical]

Gotcha, no worries! It's been working really well paired with ComicbookArchiveToolbox: Merge original files -> Split into 1.99GB files with CBAT -> Resize and convert those with ACBR -> Remerge newly compressed parts into one -> fin

[binarynonsense]

Nice, glad you've found a way to do what you wanted and a use for the tool despite its current problems/limitations. I'll let you know here if I some day manage to make a new version that can handle these big files.

And thanks again for taking the time to add more info to the issue and make tests to help me narrow down the cause.

edit: fixed some typos and minor rewording.

[binarynonsense]

I've uploaded a new beta (3.3.3-beta4) of ACBR that adds support for reading, converting, extracting and creating CBZ, CBR and CB7 files bigger than 2GB (to create CBR files you need to configure an external third party tool as explained in the docs).

So far my tests have been positive, but let me know if it works for you if you have the time to try it.

@tactiical the problem you had with the 12GB file, which was different than just the size of the file, should probably be also solved but I don't have a file that big to test :) (I had the same problem with one of my bigger test files and it now works after I changed some things).

PS: I won't be updating ACBT in the foreseeable future so, for now at least, ACBR will be the safer bet of the two when it comes to dealing with big files.