ACE archive support

[wesinator]

Would it be possible to support extracting ACE format archives ?

Thanks,

[gingerbeardman]

Reference for @aonez https://pypi.org/project/acefile/

[aonez]

Already tried to port the old PPC source a few months ago, very unstable. Will take a look at the python project 👍🏼

[Darklocq]

I've tested the python project (see https://www.roe.ch/acefile for more detailed info) extensively, over two years, and can no longer find any ACE archives (including the ACE 2.0 extensions by WinZip) that it will not properly extract. I think I tracked down the last problematic variant, reported it, and the dev fixed it, about a year and a half ago.

Having a convenient Mac tool for this format is important enough to bother with, because many mods for The Elder Scrolls III: Morrowind are packed in that format, and Morrowind has had a whole lot of new life breathed into it by the release a couple of years ago of a complete and stable version of OpenMW (open-source, cross-platform engine for the game, supporting graphics modernization, and which in turn is the base of the multi-player versions). While you can use the Python script if you're a super-nerd and have set everything up just right, the average gamer isn't commandline-friendly, and a GUI extractor that works on modern MacOS is needed. It would also be very handy for extracting other older Windows stuff (e.g. PC libraries of models and textures that will also work in Mac versions of Poser and other 3D apps). I doubt there is any call for creating ACE archives, though. Even Windows people have abandoned that format over a decade ago.

[aonez]

@Darklocq maybe porting that to C/C++. It's hard to run a python script in a sandboxed environment...

[gingerbeardman]

I'm not sure how good/useful the output is, but you can compile a Python script to a binary.

https://docs.python-guide.org/shipping/freezing/

Most likely: https://www.pyinstaller.org

[aonez]

Yeah, already did that... but the onefile method crashes anyway in sandbox (expands some libraries and signature validation fails) and without that option the files needed weight about 10MB. Anyway I'll be posting a build version in a few hours.

[aonez]

Error loading Python lib '/var/folders/3s/_rmgzhd12qx35mtpq4b62kmh0000gn/T/com.aone.keka/_MEIZ4mVsV/Python': 
...
(/var/folders/3s/_rmgzhd12qx35mtpq4b62kmh0000gn/T/com.aone.keka/_MEIZ4mVsV/Python)
not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed."

[aonez]

Here a test build:

https://github.com/aonez/Keka/releases/download/dev-test-builds/Keka-ACE.7z

[Darklocq]

The test build worked for me (including a solid-archive Ace 2.x file that had formerly been problematic in an earlier version of acefile.py).

The test build did crash on me once, when I was using Preferences (after opening New Window); when I switched to the Extraction pane, the app died. I wasn't able to repro the crash. Maybe it was something to do with updating the prefs file from a non-beta version.

[aonez]

The test build did crash

@Darklocq In the Console.app, in User Reports you can find a Keka_*.crash. If you can share that file would be awesome.

[Darklocq]

It actually made .cpu_resources.diag and .hang files. I don't have the originals from the other day (I clear out logs pretty quickly), but it was easy to generate more; here's a 7z of several: https://mega.nz/#!xBZWQa7Z!Z_PnmDA5lWQAl2pGI4WbFgv501m0jUqzIXVHjOYJQlA

None of these had anything to do with working with ACE files in particular. The main ways of generating them were:

1. Starting a long operation (compression to 7z, max, in my case) and switching out of the app to go do something else, which produced a hang that required a force-quit.
2. Starting a long operation then pressing the [x] widget in the progress indicator to cancel the operation; also produced a force-quit hang.
3. Starting a compression in a directory for which the user doesn't have write access; produced an outright crash.

I think the other day I produced a full-on crash by some other means, but I don't recall exactly what I was doing.

I have not tried one of the other recent betas aside from the one with the new ACE 2.x code, so I don't know if these issues relate to that code in some way despite not doing ACE-specific operations.

[aonez]

Thanks a lot @Darklocq will check that tomorrow.

[aonez]

@Darklocq check the latest dev build: https://github.com/aonez/Keka/releases/tag/v1.2.0-dev.3742

I've fixed an issue only present in 10.13 that was causing the hang. Let me know if it still hangs in your configuration.

[Darklocq]

Dev 3742 still gives me the hanging problem. It works fine until you switch away from the app, then Keka becomes unresponsive. It still SOMETIMES actually finishes the compression job it was on, but does it very slowly, and without any indication when it is finished. If it's a big job, it will not finish, and instead will use up more and more memory until either you kill it or the system crashes.

I watched file size on one archive (with a known size at completion, from having compressed it before with the release version of Keka). The dev version created the file and started saving to it incrementally. I switched away from the app, and then Keka seemed to be dead, but actually continued compressing some of the material and was writing that part out continually. Around the 140 MB mark, the OS spat out a cpu_resource.diag file about Keka. The app continued in half-functional fashion (still compressing silently but with an unresponsive GUI). When the file got to about 230 MB, Keka just stopped dead. It went into what looked like idleness for another 20+ minutes (and showed up as "not responding" in Activity Monitor), after which one of my system monitors (CleanMyMac, I think) gave me a low memory warning. I looked in iStat menus, and Keka dev was using the bulk of the memory, and it kept climbing (several % per minute). Act. Mon. showed it using more and more real, virtual, and private memory each polling cycle; also using almost 100% of at least one CPU core. So, I force-quit Keka dev to prevent another kernel panic. I repeated this test with essentially the same results, though the numbers I think were a bit different.

Regardless, the dev version remains unresponsive during and after the compression you switched away from to do something else, until you force-quit it, even if the file is small enough to complete before Keka locks up totally and sucks up all the RAM and virtual memory.

Attached are two of the .cpu_resource.diag logs from today, and one from the other day when I first tried dev 3742. Also included is a .core_analytics generated just minutes after the first of these .cpu_resource.diag files, so possibly related.

Keka dev 3742 logs.zip

[aonez]

~This will come in 1.2.0. No Apple Silicon support yet.~

Thanks @gingerbeardman and @Darklocq for all the suggestions and feedback 🤎

Can't make it work properly.