Closed GoogleCodeExporter closed 8 years ago
Original comment by sknau...@wesleyan.edu
on 9 May 2008 at 12:56
I would like to see this too.
Original comment by sgorman07@gmail.com
on 21 May 2008 at 1:52
Could lzma file support be added as well, please?
Original comment by tenebra...@gmail.com
on 24 May 2008 at 4:37
Lzma is the default compression type of 7zip. If this is a separate file format
and
not just an extension used for 7zip files (.7zX is a common Mac OS extension,
.lzma
seems to be also used), more information would be needed.
Ideally one would also look into if the 7zip sdk can handle it even if it were
a
separate file format.
Original comment by sknau...@wesleyan.edu
on 25 May 2008 at 5:53
I realise LZMA is the default codec in 7z files, but lzma is an extension as
well ;)
It's similar to bz2 and gz files, a minimal compression format with no archiving
features. It's gradually gaining ground on bz2, for instance libpng source
tarballs
are now compressed with lzma, as is GNU's coreutils.
I believe the SDK can extract lzma files, based on the fact lzma support was
added to
7-Zip 4.58. There was and still is a package for handling lzma files:
http://tukaani.org/lzma
Original comment by tenebra...@gmail.com
on 25 May 2008 at 11:50
Hey, sorry for the delay. I recently took a more in-depth look at the SDK and
the
outlook is not good. 7zip archives are just not built to extract one file at a
time
or small parts of files (like the first 4 bytes mupen uses to tell if the
decompressed file is a ROM or not). This would mean that we would need to
decompress an entire archive to test if it contains ROMs and decompress the
entire
(or at least most of the) archive to access any ROMs in it. Igor Pavlov (the
7zip
author) recently confirmed what I discovered by looking at the code.
This seems to suggest 7zip is a poor archive format for direct use with mupen.
I
got bzip2 support in my local copy, waiting on Richard to okay RCS before I
merge
that, and direct lzma compression should also be okay. Its a bit of a pain, but
moving your 7zips to either format should take up roughly the same amount of
space
but make it much easier for directly using the compressed files.
Since this is tied for 2nd as our most requested feature, I think we might want
to
try a little more before writing it off, though when the guy who designed the
filetype tells you its not possible, that's fairly discouraging.
Original comment by sknau...@wesleyan.edu
on 18 Jun 2008 at 6:25
Just as a note. 7zips are VERY good when zipping multiple, similar roms (like
all the
different versions of a game). I have a 16.1 MB 7z archive with many versions of
Super Smash Bros. Decompressed it contains 21 roms and is 378.9 MB. The ROM I
use is
16 MB. Those files in a bzip2 archive is 243.7 MB.
On my amd64 laptop using File Roller and a stop-watch: running at 800 MHz it
took 11
sec to get the rom I want; running at 1.8 GHz it took 5 sec to get the rom I
want.
The bzip2 version of the archive was unusable; it takes minutes to compress and
minutes to just get a file listing in File Roller. The 7z doesn't even take a
second
for the file listing.
I have no real problem extracting the file that I want from the 7zip before
using
mupen64plus; however, even if it is less than perfect, it would be nice for
7zip support.
Original comment by ejon...@gmail.com
on 19 Jun 2008 at 3:26
Okay, I did some tests and found I was very wrong about the size. 7zip wins
hands
down for compression for multiple files due to the large sliding dictionary. As
most ROMs with multiple versions share a lot of code, this lets 7zip's LZMA do
essentially binary diffs on them resulting in massive (<5%)compression. LZMA,
bzip2, gzip, and zip are all on par for single files (~65-75%).
While zip multiformat archives are possible, since zip compresses each file
individually, the result is more or less the same as a bunch of individual
files.
Sadly it is this very multifile scheme which makes integrating 7zip with our
current decompression routines very difficult. However, after some though I now
think 7zip support is actually possible (and with this new information, much
more
worthwhile), it just may require a lot of rewriting of our decompression scheme
so
the system is aware of the entire archive memory space if a 7zip file is
encountered.
Meanwhile, I submitted r649 which should halve bzip2 and gzip decopression
times. I
find it does roughly halve bzip2 times, but seems to shorten gzip by
considerably
more. Gzipped ROMs now load almost as fast as uncompressed ones.
Original comment by sknau...@wesleyan.edu
on 20 Jun 2008 at 6:15
LZMA support in svn as of r659.
Original comment by sknau...@wesleyan.edu
on 21 Jun 2008 at 4:50
I can't seem to get this running. While bzip2 support runs fine, feeding mupen
a ROM
packed via "7z a asdf.lzma rom.z64", won't do a thing. Same for packing a
single ROM
with "7z a asdf.lzma -t7z -m0=lzma -mx=9 -mfb=64 -md=32m rom.z64".
Maybe I'm using wrong parameters or even the wrong app? Enlighten me, please.
rev668
Original comment by shinydo...@gmail.com
on 23 Jun 2008 at 12:05
I think "LZMA support in svn as of r659" refers to files compressed gzip-style
with
an LZMA stream compressor, not single-file 7zip archives compressed with p7zip.
If
that's the case, give the "lzma" tool a try.
Original comment by stephan....@gmail.com
on 23 Jun 2008 at 2:16
Real 7zip support is coming, I promise. Right now the LZMA support in trunk
refers
to that requested in comment #5 based on the tools "lzma" and source at
http://tukaani.org/lzma.
Original comment by sknau...@wesleyan.edu
on 23 Jun 2008 at 6:02
Oh, indeed. Looking at the sourcecode, I saw that romextension didn't cover
".7z".
Sorry for the confusion and keep us posted. :)
Original comment by shinydo...@gmail.com
on 23 Jun 2008 at 4:59
Since you asked to keep you posted:
Before adding real 7zip support I need to split archive support (zip and 7zip)
from
single file (uncompressed, bzip2, lzma, gzip) and move the code which fills a
cache
entry into a separate function so we can loop over the ROMs in an archive. I
also
need to change the rom loading code, we have one function open_rom() which I
want
to remove that basically wraps another function rom_read() and the API needs to
have an file in archive variable so the rom loading code knows which file in
the
archive to load. Ideally, an --archivefile command line option also needs to be
added for full nogui support.
The easiest way to do this, is actually to first add zip archive multirom
support,
since before we stopped at the first ROM in the archive. I actually have that
mostly working in my local branch. The rom cache system can completely see the
ROMs
inside the .zip but right now since I ran into some issues changing the
open_rom()/rom_read() API, I can't actually play any but the first ROM in the
archive.
Once I work this out, I can more or less plug in the modified example 7zip SDK
decoding code I've been playing with for the past weeks and get 7zip support. I
believe that in a few days, barring unforeseen circumstances, we might be able
to
close this issue. :)
Original comment by sknau...@wesleyan.edu
on 25 Jun 2008 at 6:24
Multifile .zip support in r692.
Original comment by sknau...@wesleyan.edu
on 27 Jun 2008 at 2:29
First of all, keep up the good work! Nevertheless, I feel like having to report
the
following issues:
In r778 there seems to be some kind of memory problem when using 7z archives.
When
opening the first ROM and everything usually works great. But if I try and open
up
another ROM, I get the following message in my console:
"Deflating 7zip archive.
7zip decoder can not allocate memory
Error: Couldn't load Rom!"
I can only fix this by restarting the emulator itself. The number of ROMs I can
open
varies drastically. Maybe 7z is given a limited amount of memory that is not
set free
again after (successfully) extracting?
Something different: when you close mupen while it's scanning your ROM
directories,
the progress is not saved until it's finished and thus you'd again have to go
through
scanning everything until it finishes. This could become a problem if you
accidentally close mupen or if it crashes for some reasong while scanning. For
crashing while scanning: why not print out the file it's working on right now?
This
would make debugging easier instead of having to guess which file is the one to
blame.
Original comment by shinydo...@gmail.com
on 15 Jul 2008 at 11:33
The 7zip code has known memory issues. This is basically that the simplified C
decoder in the SDK listed above allocates in single large blocks. This works
fine
for small 7zip archives, but if the uncompressed size of the 7z archive is
close to
the total system memory, it will fail. Since doing things with the emulator
allocates memory, its very possible that you cross this limit during runtime.
My
PIII, while slow, has 768MB of RAM and I only encountered this error on the
Zelda
GoodName archives, which have uncompressed sizes of 1.5GB and 2.2GB
respectively or
if I tried to load a 7zip ROM while the RCS thread is processing a 7zip ROM.
I encountered the variation in decoder behavior and spent a whole day
Valgrinding
the GUI. There is no memory leak from 7zip support. I suspect this has to do
with
allocations and frees making it impossible to load continuous blocks of memory
even
if the memory is there.
The poor memory management of the C code was not mentioned in the
documentation,
and the C++ code (basically which is the 7zr utility) is both quite large (250+
files, 600KB compiled, 2/3 of mupen's core) and extremely poorly written. Igor
Pavlov has also refused to explain what algorithm the C++ code uses to allocate
memory in a better way (It must decompress and flush the previous file, but the
LZMA stream crosses file boundaries, how exactly the decoder is cached is
non-trivial), further hindering my progress.
As far as the bottom issue, there is a commented out line romcache.c 394 which
does
just that which I enable when debugging.
Original comment by sknau...@wesleyan.edu
on 15 Jul 2008 at 2:59
PS - The memory problem is why this issue hasn't been closed. I'm working on
it.
However, my vision of just dropping in library code (like I do for gzip, zip,
bzip2, and lzma) doesn't seem possible. I'm trying to just modify the current
source, but I'm almost at the point of just rewriting it. The lack of any
comments
(even something vague like what the function does) is really frustrating.
Compare
say my main/rom.c with main/7zip/7zExtract.c to see the difference.
Original comment by sknau...@wesleyan.edu
on 15 Jul 2008 at 3:09
I can't fully verify the memory limit crossing. The Public Domain Goodname
archive is
about 150 MB unpacked for me (I threw out o, b, h, t and f ROMs). If I try to
open
"Pip's Pong my Mr. Pips (PD)", I get just the same error message. I have 1280
MB of
RAM in total and about 450 MByte of free space. So maybe I'm getting this error
due
to not enough continuous memory being available?
Thanks for the hint about showing which is being processed on start up.
Original comment by shinydo...@gmail.com
on 15 Jul 2008 at 4:32
I feel like digging this up again as I just ran into another strange thing. I
set the
ROM directory to the one on my second HDD where I keep my 7zipped ROMs. Once the
whole ROM cache was created and I restart mupen, it immediately rescans some of
the
files for no particular reason. From time to time, it seems that the cache was
emptied and has to be rebuilt from scratch. Not only does this take a long
time, it's
also pointless if the files didn't change. I can only guess that this is caused
by
closing mupen when the few files that have to be rechecked weren't fully
indexed yet.
Again, what are the criteria for rechecking a file?
Another thing I just noticed is that when clearing a directory from the ROM
browser,
the current scanning process running in the background is not stopped. This
happens
regardless of the files being scanned being 7zipped or regularly unpacked.
Any progress on forcing to write to the ROM cache when mupen is being closed or
just
every x ROMs to save time at the next start?
Original comment by shinydo...@gmail.com
on 21 Sep 2008 at 6:13
When using .zip archive, the calculation of the MD5 fails (is wrong). Which
means
that the "goodname" will not be inserted to the dB/GUI. If I manually unzip the
file
first, the MD5 calculation is correct. I have only 1 .z64 in each archive.
The game is however working.
I'm running trunk r1095.
I don't know if it is relevant, but the in file 'main/rom.c' function
'load_archive_rom', the unzGetCurrentFileInfo function call returns -1.
I have located it to be this specific UNZ_ERRNO that is returned.
if(lSeek!=0)
{
if(ZSEEK(s->z_filefunc, s->filestream,lSeek,ZLIB_FILEFUNC_SEEK_CUR)==0)
lSeek=0;
}
else
err=UNZ_ERRNO;
Original comment by olejl77@gmail.com
on 10 Oct 2008 at 8:56
I can confirm this for r1096.
If I try and load a ROM directly from command line, mupen exits with
"main/rom.c,
347: Out of memory!". If I open the ROM from the ROM list, it loads (name and
manufacturer are correct on the terminal, somehow; yet the size is down to
40something KB), but it doesn't start the game. glN64 opens its window, Glide
triggers a SIGSEGV for the core and Rice sometimes freezes the whole emulator
with
its last message being "Xlib: unexpected async reply (sequence 0x5a)!".
Original comment by shinydo...@gmail.com
on 10 Oct 2008 at 5:39
Guys, please don't put new unrelated issues in old reports.
I believe I tracked the game fails to detect as a .zip to a misplaced } when I
tried to fix some -Wall warnings in the library around r1060. Actually in the
above
code, the } should be after err=UNZ_ERRNO and not before the else. This caused
the
reported the size of any read to be 0, causing a host of other symptoms. As of
r1097 I can see .zip compressed ROMs again and open them from the commandline
just
fine.
Original comment by sknau...@wesleyan.edu
on 11 Oct 2008 at 2:56
Yes this solves the problem I described in comment #21.
Original comment by olejl77@gmail.com
on 11 Oct 2008 at 5:43
please add support for this, it would save a lot of people a lot of space
Original comment by Legato...@gmail.com
on 27 Jul 2009 at 7:53
Could someone add LZMA2 (.xz) format support please? It's set to replace LZMA
(.lzma).
Original comment by tenebra...@gmail.com
on 29 Jul 2009 at 2:09
This enhancement request is invalid given the new architecture of Mupen64Plus.
The
core library only accepts uncompressed ROMs, so the task of de-compressing any
zipped
ROMs lies with the front-end application. The console-UI also only currently
supports uncompressed ROMs as well.
Original comment by richard...@gmail.com
on 11 Jan 2010 at 3:39
Original issue reported on code.google.com by
sknau...@wesleyan.edu
on 9 May 2008 at 12:53