Add support for .7z 7zip compressed ROMs

[GoogleCodeExporter]

Should be fairly straightforward. In the SDK at 
http://www.7-zip.org/sdk.html is sample C code for opening an archive. 
After Issue 66 is taken care of, it should be fairly clear where to add 
code to deal with compressed roms in rom.c. 

I'd also suggest as part of this issue, that we put the zip library and 
the 7zip libraries into a folder, such as /main/compression/ as nothing 
else but the rom code currently uses it (I believe) and this makes it 
easier to keep track of. 

I'll take care of this, but Issue 66 and the general rom cache system are 
on my TODO first. Just making an issue since this was a requested feature 
in case someone wants to tackle it before me.

Original issue reported on code.google.com by sknau...@wesleyan.edu on 9 May 2008 at 12:53

[GoogleCodeExporter]

Original comment by sknau...@wesleyan.edu on 9 May 2008 at 12:56

• Added labels: Priority-Medium
• Removed labels: Priority-Low

[GoogleCodeExporter]

I would like to see this too.

Original comment by sgorman07@gmail.com on 21 May 2008 at 1:52

[GoogleCodeExporter]

Could lzma file support be added as well, please?

Original comment by tenebra...@gmail.com on 24 May 2008 at 4:37

[GoogleCodeExporter]

Lzma is the default compression type of 7zip. If this is a separate file format 
and 
not just an extension used for 7zip files (.7zX is a common Mac OS extension, 
.lzma 
seems to be also used), more information would be needed.

Ideally one would also look into if the 7zip sdk can handle it even if it were 
a 
separate file format.

Original comment by sknau...@wesleyan.edu on 25 May 2008 at 5:53

[GoogleCodeExporter]

I realise LZMA is the default codec in 7z files, but lzma is an extension as 
well ;)
It's similar to bz2 and gz files, a minimal compression format with no archiving
features. It's gradually gaining ground on bz2, for instance libpng source 
tarballs
are now compressed with lzma, as is GNU's coreutils.

I believe the SDK can extract lzma files, based on the fact lzma support was 
added to
7-Zip 4.58. There was and still is a package for handling lzma files:
http://tukaani.org/lzma

Original comment by tenebra...@gmail.com on 25 May 2008 at 11:50

[GoogleCodeExporter]

Hey, sorry for the delay. I recently took a more in-depth look at the SDK and 
the 
outlook is not good. 7zip archives are just not built to extract one file at a 
time 
or small parts of files (like the first 4 bytes mupen uses to tell if the 
decompressed file is a ROM or not). This would mean that we would need to 
decompress an entire archive to test if it contains ROMs and decompress the 
entire 
(or at least most of the) archive to access any ROMs in it. Igor Pavlov (the 
7zip 
author) recently confirmed what I discovered by looking at the code. 

This seems to suggest 7zip is a poor archive format for direct use with mupen. 
I 
got bzip2 support in my local copy, waiting on Richard to okay RCS before I 
merge 
that, and direct lzma compression should also be okay. Its a bit of a pain, but 
moving your 7zips to either format should take up roughly the same amount of 
space 
but make it much easier for directly using the compressed files.

Since this is tied for 2nd as our most requested feature, I think we might want 
to 
try a little more before writing it off, though when the guy who designed the 
filetype tells you its not possible, that's fairly discouraging.

Original comment by sknau...@wesleyan.edu on 18 Jun 2008 at 6:25

[GoogleCodeExporter]

Just as a note. 7zips are VERY good when zipping multiple, similar roms (like 
all the
different versions of a game). I have a 16.1 MB 7z archive with many versions of
Super Smash Bros. Decompressed it contains 21 roms and is 378.9 MB. The ROM I 
use is
16 MB. Those files in a bzip2 archive is 243.7 MB.

On my amd64 laptop using File Roller and a stop-watch: running at 800 MHz it 
took 11
sec to get the rom I want; running at 1.8 GHz it took 5 sec to get the rom I 
want.

The bzip2 version of the archive was unusable; it takes minutes to compress and
minutes to just get a file listing in File Roller. The 7z doesn't even take a 
second
for the file listing.

I have no real problem extracting the file that I want from the 7zip before 
using
mupen64plus; however, even if it is less than perfect, it would be nice for 
7zip support.

Original comment by ejon...@gmail.com on 19 Jun 2008 at 3:26

[GoogleCodeExporter]

Okay, I did some tests and found I was very wrong about the size. 7zip wins 
hands 
down for compression for multiple files due to the large sliding dictionary. As 
most ROMs with multiple versions share a lot of code, this lets 7zip's LZMA do 
essentially binary diffs on them resulting in massive (<5%)compression. LZMA, 
bzip2, gzip, and zip are all on par for single files (~65-75%).

While zip multiformat archives are possible, since zip compresses each file 
individually, the result is more or less the same as a bunch of individual 
files.

Sadly it is this very multifile scheme which makes integrating 7zip with our 
current decompression routines very difficult. However, after some though I now 
think 7zip support is actually possible (and with this new information, much 
more 
worthwhile), it just may require a lot of rewriting of our decompression scheme 
so 
the system is aware of the entire archive memory space if a 7zip file is 
encountered.

Meanwhile, I submitted r649 which should halve bzip2 and gzip decopression 
times. I 
find it does roughly halve bzip2 times, but seems to shorten gzip by 
considerably 
more. Gzipped ROMs now load almost as fast as uncompressed ones.

Original comment by sknau...@wesleyan.edu on 20 Jun 2008 at 6:15

[GoogleCodeExporter]

LZMA support in svn as of r659.

Original comment by sknau...@wesleyan.edu on 21 Jun 2008 at 4:50

[GoogleCodeExporter]

I can't seem to get this running. While bzip2 support runs fine, feeding mupen 
a ROM
packed via "7z a asdf.lzma rom.z64", won't do a thing. Same for packing a 
single ROM
with "7z a asdf.lzma -t7z -m0=lzma -mx=9 -mfb=64 -md=32m rom.z64".
Maybe I'm using wrong parameters or even the wrong app? Enlighten me, please.
rev668

Original comment by shinydo...@gmail.com on 23 Jun 2008 at 12:05

[GoogleCodeExporter]

I think "LZMA support in svn as of r659" refers to files compressed gzip-style 
with 
an LZMA stream compressor, not single-file 7zip archives compressed with p7zip. 
If 
that's the case, give the "lzma" tool a try.

Original comment by stephan....@gmail.com on 23 Jun 2008 at 2:16

[GoogleCodeExporter]

Real 7zip support is coming, I promise. Right now the LZMA support in trunk 
refers 
to that requested in comment #5 based on the tools "lzma" and source at 
http://tukaani.org/lzma.

Original comment by sknau...@wesleyan.edu on 23 Jun 2008 at 6:02

[GoogleCodeExporter]

Oh, indeed. Looking at the sourcecode, I saw that romextension didn't cover 
".7z".
Sorry for the confusion and keep us posted. :)

Original comment by shinydo...@gmail.com on 23 Jun 2008 at 4:59

[GoogleCodeExporter]

Since you asked to keep you posted:

Before adding real 7zip support I need to split archive support (zip and 7zip) 
from 
single file (uncompressed, bzip2, lzma, gzip) and move the code which fills a 
cache 
entry into a separate function so we can loop over the ROMs in an archive. I 
also 
need to change the rom loading code, we have one function open_rom() which I 
want 
to remove that basically wraps another function rom_read() and the API needs to 
have an file in archive variable so the rom loading code knows which file in 
the 
archive to load. Ideally, an --archivefile command line option also needs to be 
added for full nogui support. 

The easiest way to do this, is actually to first add zip archive multirom 
support, 
since before we stopped at the first ROM in the archive. I actually have that 
mostly working in my local branch. The rom cache system can completely see the 
ROMs 
inside the .zip but right now since I ran into some issues changing the 
open_rom()/rom_read() API, I can't actually play any but the first ROM in the 
archive. 

Once I work this out, I can more or less plug in the modified example 7zip SDK 
decoding code I've been playing with for the past weeks and get 7zip support. I 
believe that in a few days, barring unforeseen circumstances, we might be able 
to 
close this issue. :)

Original comment by sknau...@wesleyan.edu on 25 Jun 2008 at 6:24

[GoogleCodeExporter]

Multifile .zip support in r692.

Original comment by sknau...@wesleyan.edu on 27 Jun 2008 at 2:29

[GoogleCodeExporter]

First of all, keep up the good work! Nevertheless, I feel like having to report 
the
following issues:

In r778 there seems to be some kind of memory problem when using 7z archives. 
When
opening the first ROM and everything usually works great. But if I try and open 
up
another ROM, I get the following message in my console:
"Deflating 7zip archive.
7zip decoder can not allocate memory
Error: Couldn't load Rom!"
I can only fix this by restarting the emulator itself. The number of ROMs I can 
open
varies drastically. Maybe 7z is given a limited amount of memory that is not 
set free
again after (successfully) extracting?

Something different: when you close mupen while it's scanning your ROM 
directories,
the progress is not saved until it's finished and thus you'd again have to go 
through
scanning everything until it finishes. This could become a problem if you
accidentally close mupen or if it crashes for some reasong while scanning. For
crashing while scanning: why not print out the file it's working on right now? 
This
would make debugging easier instead of having to guess which file is the one to 
blame.

Original comment by shinydo...@gmail.com on 15 Jul 2008 at 11:33

[GoogleCodeExporter]

The 7zip code has known memory issues. This is basically that the simplified C 
decoder in the SDK listed above allocates in single large blocks. This works 
fine 
for small 7zip archives, but if the uncompressed size of the 7z archive is 
close to 
the total system memory, it will fail. Since doing things with the emulator 
allocates memory, its very possible that you cross this limit during runtime. 
My 
PIII, while slow, has 768MB of RAM and I only encountered this error on the 
Zelda 
GoodName archives, which have uncompressed sizes of 1.5GB and 2.2GB 
respectively or 
if I tried to load a 7zip ROM while the RCS thread is processing a 7zip ROM.

I encountered the variation in decoder behavior and spent a whole day 
Valgrinding 
the GUI. There is no memory leak from 7zip support. I suspect this has to do 
with 
allocations and frees making it impossible to load continuous blocks of memory 
even 
if the memory is there.

The poor memory management of the C code was not mentioned in the 
documentation, 
and the C++ code (basically which is the 7zr utility) is both quite large (250+ 
files, 600KB compiled, 2/3 of mupen's core) and extremely poorly written. Igor 
Pavlov has also refused to explain what algorithm the C++ code uses to allocate 
memory in a better way (It must decompress and flush the previous file, but the 
LZMA stream crosses file boundaries, how exactly the decoder is cached is 
non-trivial), further hindering my progress. 

As far as the bottom issue, there is a commented out line romcache.c 394 which 
does 
just that which I enable when debugging.

Original comment by sknau...@wesleyan.edu on 15 Jul 2008 at 2:59

[GoogleCodeExporter]

PS - The memory problem is why this issue hasn't been closed. I'm working on 
it. 
However, my vision of just dropping in library code (like I do for gzip, zip, 
bzip2, and lzma) doesn't seem possible. I'm trying to just modify the current 
source, but I'm almost at the point of just rewriting it. The lack of any 
comments 
(even something vague like what the function does) is really frustrating. 
Compare 
say my main/rom.c with main/7zip/7zExtract.c to see the difference.

Original comment by sknau...@wesleyan.edu on 15 Jul 2008 at 3:09

[GoogleCodeExporter]

I can't fully verify the memory limit crossing. The Public Domain Goodname 
archive is
about 150 MB unpacked for me (I threw out o, b, h, t and f ROMs). If I try to 
open
"Pip's Pong my Mr. Pips (PD)", I get just the same error message. I have 1280 
MB of
RAM in total and about 450 MByte of free space. So maybe I'm getting this error 
due
to not enough continuous memory being available?
Thanks for the hint about showing which is being processed on start up.

Original comment by shinydo...@gmail.com on 15 Jul 2008 at 4:32

[GoogleCodeExporter]

I feel like digging this up again as I just ran into another strange thing. I 
set the
ROM directory to the one on my second HDD where I keep my 7zipped ROMs. Once the
whole ROM cache was created and I restart mupen, it immediately rescans some of 
the
files for no particular reason. From time to time, it seems that the cache was
emptied and has to be rebuilt from scratch. Not only does this take a long 
time, it's
also pointless if the files didn't change. I can only guess that this is caused 
by
closing mupen when the few files that have to be rechecked weren't fully 
indexed yet.
Again, what are the criteria for rechecking a file?

Another thing I just noticed is that when clearing a directory from the ROM 
browser,
the current scanning process running in the background is not stopped. This 
happens
regardless of the files being scanned being 7zipped or regularly unpacked.

Any progress on forcing to write to the ROM cache when mupen is being closed or 
just
every x ROMs to save time at the next start?

Original comment by shinydo...@gmail.com on 21 Sep 2008 at 6:13

[GoogleCodeExporter]

When using .zip archive, the calculation of the MD5 fails (is wrong). Which 
means 
that the "goodname" will not be inserted to the dB/GUI. If I manually unzip the 
file 
first, the MD5 calculation is correct. I have only 1 .z64 in each archive. 

The game is however working.

I'm running trunk r1095.

I don't know if it is relevant, but the in file 'main/rom.c' function 
'load_archive_rom', the unzGetCurrentFileInfo function call returns -1.

I have located it to be this specific UNZ_ERRNO that is returned.
if(lSeek!=0)
    {
    if(ZSEEK(s->z_filefunc, s->filestream,lSeek,ZLIB_FILEFUNC_SEEK_CUR)==0)
        lSeek=0;
    }
    else
        err=UNZ_ERRNO;

Original comment by olejl77@gmail.com on 10 Oct 2008 at 8:56

[GoogleCodeExporter]

I can confirm this for r1096.
If I try and load a ROM directly from command line, mupen exits with 
"main/rom.c,
347: Out of memory!". If I open the ROM from the ROM list, it loads (name and
manufacturer are correct on the terminal, somehow; yet the size is down to
40something KB), but it doesn't start the game. glN64 opens its window, Glide
triggers a SIGSEGV for the core and Rice sometimes freezes the whole emulator 
with
its last message being "Xlib: unexpected async reply (sequence 0x5a)!".

Original comment by shinydo...@gmail.com on 10 Oct 2008 at 5:39

[GoogleCodeExporter]

Guys, please don't put new unrelated issues in old reports.

I believe I tracked the game fails to detect as a .zip to a misplaced } when I 
tried to fix some -Wall warnings in the library around r1060. Actually in the 
above 
code, the } should be after err=UNZ_ERRNO and not before the else. This caused 
the 
reported the size of any read to be 0, causing a host of other symptoms. As of 
r1097 I can see .zip compressed ROMs again and open them from the commandline 
just 
fine. 

Original comment by sknau...@wesleyan.edu on 11 Oct 2008 at 2:56

[GoogleCodeExporter]

Yes this solves the problem I described in comment #21.

Original comment by olejl77@gmail.com on 11 Oct 2008 at 5:43

[GoogleCodeExporter]

please add support for this, it would save a lot of people a lot of space

Original comment by Legato...@gmail.com on 27 Jul 2009 at 7:53

[GoogleCodeExporter]

Could someone add LZMA2 (.xz) format support please? It's set to replace LZMA 
(.lzma).

Original comment by tenebra...@gmail.com on 29 Jul 2009 at 2:09

[GoogleCodeExporter]

This enhancement request is invalid given the new architecture of Mupen64Plus.  
The
core library only accepts uncompressed ROMs, so the task of de-compressing any 
zipped
ROMs lies with the front-end application.  The console-UI also only currently
supports uncompressed ROMs as well.

Original comment by richard...@gmail.com on 11 Jan 2010 at 3:39

• Changed state: Invalid