Idea: crunched executable for cartridge

[rhalkyard]

I've noticed the ongoing balancing act between adding new features, and keeping the default environment small enough to fit in a 16k cartridge. Right now it seems like we can't add much without taking away or heavily optimizing something else. I realise that is a good incentive for clean code and minimalism ;) but I do have an idea to help alleviate the pressure: crunch the program that we write to the cartridge.

It doesn't make as much sense to crunch the programs on disk since the user is likely to save their own Forth that we can't crunch (unless we add compression to SAVE-PACK...). But the cartridge version is static, and PC-based crunchers like Exomizer and Pucrunch are extremely effective, and generate self-extracting executables that don't require any code changes or user intervention.

I built cartridges from executables crunched with Exomizer and Pucrunch, and compared their size and startup time against the current uncompressed cartridge:

	Executable size	Cartridge bytes free	Startup Time
Uncompressed	16045 bytes	235 bytes	0.4s
exomizer sfx sys	9747 bytes	6533 bytes	2.2s
pucrunch -fshort	10290 bytes	5990 bytes	4.3s

Startup time was measured in VICE by setting a breakpoint at interpret_loop, resetting with the cartridge image attached, and dividing the cycle counter at the breakpoint by 1000000. NTSC machines will be fractionally faster than this, and PAL machines fractionally slower.

Exomizer looks like the winner here on both compression ratio and uncrunching speed, but in both cases we win a LOT more cartridge space, at the cost of a couple of seconds of startup time. It might be possible to speed things up a bit (and save a few more bytes) by writing a custom uncrunch routine that runs directly from the cartridge - right now the crunched executable is copied to RAM first, which is redundant.

[jkotlinski]

:-) it is something i considered too. But isn’t part of the attraction with a cartridge that it starts near-instantly? Also, I think it is good that the unpacked footprint stays within a certain limit, so that there is plenty of space for user programs.

Some C64 Forths are very feature-rich but leave little space for user code and data, it’s a trap I’d like to steer clear of :-)

On Thu, 17 Dec 2020 at 17:04, Richard Halkyard notifications@github.com wrote:

I've noticed the ongoing balancing act between adding new features, and keeping the default environment small enough to fit in a 16k cartridge. Right now it seems like we can't add much without taking away or heavily optimizing something else. I realise that is a good incentive for clean code and minimalism ;) but I do have an idea to help alleviate the pressure: crunch the program that we write to the cartridge.

It doesn't make as much sense to crunch the programs on disk since the user is likely to save their own Forth that we can't crunch (unless we add compression to SAVE-PACK...). But the cartridge version is static, and PC-based crunchers like Exomizer https://bitbucket.org/magli143/exomizer/wiki/Home and Pucrunch http://a1bert.kapsi.fi/Dev/pucrunch/ are extremely effective, and generate self-extracting executables that don't require any code changes or user intervention.

I built cartridges from executables crunched with Exomizer and Pucrunch, and compared their size and startup time against the current uncompressed cartridge: Executable size Cartridge bytes free Startup Time Uncompressed 16045 bytes 235 bytes 0.4s exomizer sfx sys 9747 bytes 6533 bytes 2.2s pucrunch -fshort 10290 bytes 5990 bytes 4.3s

Startup time was measured in VICE by setting a breakpoint at interpret_loop, resetting with the cartridge image attached, and dividing the cycle counter at the breakpoint by 1000000. NTSC machines will be fractionally faster than this, and PAL machines fractionally slower.

Exomizer looks like the winner here on both compression ratio and uncrunching speed, but in both cases we win a LOT more cartridge space, at the cost of a couple of seconds of startup time. It might be possible to speed things up a bit (and save a few more bytes) by writing a custom uncrunch routine that runs directly from the cartridge - right now the crunched executable is copied to RAM first, which is redundant.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jkotlinski/durexforth/issues/307, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY34O7K4ME7I3IB7KSMMADSVIT2PANCNFSM4U74C36Q .

[rhalkyard]

Fair enough! Just been thinking about cartridge size a lot while playing with IO code for #252 - it will cost a few bytes to replace OPENW and CLOSEW with a more flexible set of words, and if V depends on them, they can't be optional. I'll keep thinking :)

[burnsauce]

Keep in mind, I just fleshed out a bunch of features for V. Not all of these features are particularly useful, and I wouldn't object to trimming some of them.

[Kroc]

What actually occupies the most space on the cart? Perhaps some rejigging of data-structures could save space, or selective compression of text, for example, without resorting to blindly compressing the entire cart.

[gitjeff2]

Another thought occurs to me: the Jupiter Ace (Z80) implemented a Forth in well under 8kB of ROM. Unfortunately, the system shipped with paltry 3kB of RAM of which only about 1kB was available to the user. The 48k RAM expander required to make the system actually useful cost extra. Needless to say, the system was a flop!

I'm sure the system ROM has been dumped and commented by someone at this point. It might be worth seeing if there are any interesting optimizations — space saving and otherwise — that are reusable across architectures.

[Kroc]

That would probably be an unwise legally as the rights to the Jupiter Ace were bought up by a company known for going after people. The Ace's Forth was compact because it compiled words to assembly and actually disassembled code to restore it to terms.

[gitjeff2]

That would probably be an unwise legally as the rights to the Jupiter Ace were bought up by a company known for going after people.

@Kroc, you concern is entirely valid. To be slightly more clear, I wasn't referring to duplicating copyright encumbered code. That would be bad for exactly the reasons you allude to. Rather, I was referring to general implementation details that could be abstracted and reproduced without directly relying on the Jupiter Ace ROM code itself.

But, it sounds like you're saying the way way the Jupiter Ace handles Forth is very different from that of DurexForth such that they wouldn't have much in common. Fair enough.