Metal Gear Solid Master Collection

[jampash1996]

Unable to extract anything from MGS1 master collections alldata.psm.m / alldata.bin Checked the exe in HxD and found the seed - 25G/xpvTbsb+6

When running MArchiveBatchTool.exe fullunpack --keep alldata.psb.m zlib 25G/xpvTbsb+6 64 it gives the following error

Error while processing: Newtonsoft.Json.JsonReaderException: Could not convert to integer: 2289213440. Path 'file_info['system/roms/MGS_EN_DISC1-washed.BIN'][0]'. ---> System.OverflowException: Value was either too large or too small for an Int32.

[Keith-Bateman]

I suspect that this is related to an issue with the MArchiveBatchTool that was found in the last year. If you go to the official GitLab (https://gitlab.com/modmyclassic/sega-mega-drive-mini/marchive-batch-tool/-/tree/master/MArchiveBatchTool) then you'll notice they pushed a commit to "allow files >4GB". I'm not familiar enough to compile the project, but that's likely to solve this problem. The version of the BatchTool in this repo is from 2020, so definitely doesn't incorporate the fix.

[hadess]

The MArchiveBatchTool you link to is now a library called GMWare.M2.

Either somebody will have to write a front-end to it, or they could attempt to compile my "fork": https://github.com/masible/marchive-batch-tool

A Linux x86-64 build is available under: https://github.com/masible/project-simplunar/tree/master/bin

[petmac]

I compiled your fork and it seemed to extract the files fine. Thanks for updating it!

I submitted a PR so people can hopefully get prebuilt binaries for Windows, Linux and Mac if they want.

Unfortunately although I could extract the ROMs I'm not sure what format they're in or how to get them into something more common like bin and cue.

Listing of system/roms:

MGS_DE_DISC1-washed.BIN
MGS_DE_DISC2-washed.BIN
MGS_EN_DISC1-washed.BIN
MGS_EN_DISC2-washed.BIN
MGS_ES_DISC1-washed.BIN
MGS_ES_DISC2-washed.BIN
MGS_FR_DISK1-washed.BIN
MGS_FR_DISK2-washed.BIN
MGS_IT_DISC1-washed.BIN
MGS_IT_DISC2-washed.BIN
MGS_SPECIAL_MISSIONS-washed.BIN
mgs_de_disc1-20230407.lz4a
mgs_en_disc1-20230407.lz4a
mgs_es_disc1-20230407.lz4a
mgs_fr_disc1-20230407.lz4a
mgs_integral_disc1-20230407.lz4a
mgs_integral_disc3-20230407.lz4a
mgs_it_disc1-20230407.lz4a
mgs_jp_disc1-20230407.lz4a
mgs_sm-20230408.lz4a
mgs_us_disc1-20230407.lz4a
mgs_vrm-20230407.lz4a

[petmac]

I tried creating cue files for the BINs, like so:

MGS_EN_DISC1-washed.cue:

FILE "MGS_EN_DISC1-washed.BIN" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00

Repeat and fix the names for DISC2.

Then I created an .m3u file to allow Mednafen (via OpenEmu on Mac) to open the game.

MGS_EN.m3u:

MGS_EN_DISC1-washed.cue
MGS_EN_DISC2-washed.cue

It seems to be recognised as MGS, and starts, but gets stuck at the PlayStation logo.

[hadess]

There's a whole lot of information about M2's disc image format for the Megadrive Mini 2: https://twitter.com/GMMan_BZFlag/status/1589100536028352514

So if the MGS Master Collection disc image handling is anything like the Megadrive Mini's, then we're pretty far from stock disc images. You could probably check the header to see what format it advertises.

[OddOttAllen]

I tried creating cue files for the BINs, like so:

MGS_EN_DISC1-washed.cue:

FILE "MGS_EN_DISC1-washed.BIN" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00

Repeat and fix the names for DISC2.

Then I created an .m3u file to allow Mednafen (via OpenEmu on Mac) to open the game.

MGS_EN.m3u:

MGS_EN_DISC1-washed.cue
MGS_EN_DISC2-washed.cue

It seems to be recognised as MGS, and starts, but gets stuck at the PlayStation logo.

i have an idea as to why this is.

it has to do with the BIOS verifying it as a PS1 game. i checked with some friends and it was a bit similar to the Ecco situation... we could see by comparing the game to a retail copy that a lot of the stuff the BIOS uses to verify if the game is real or not was taken out and put onto a separate file- we're hypothesizing for now that it's the .lz4a files. therefore, hanging at the PS logo.

i'm not an expert in this by any means so feel free to correct me, but i think the collection uses this method to skip a PS1 BIOS boot up? and since the BIOS on hardware/in emulators can't see this without it being put back in, it can't get into the game

[ImanCol]

Update MGS1

I will leave a little information about what I have been able to find out about this emulator.

The .BIN ROMs of each version of the game have the empty executable [SLES-SLUS] and since this game has 2 executables (One of them is in the MGS folder within the ROM) they are still empty. In the lz4a file it has lz4 compression after the 0x2D bytes, it is a very specific type of compression, even after decompressing, it has another layer of compression for other data (before the game executable). This lz4a file contains the executable from the MGS folder contained in the ROM. The emulator verifies in the executable whether the 0x10 header (4 bytes) contains correct code and applies the specific configurations to that version. From here it becomes difficult to run other games, because they are not listed and the emulator will remain in a loop or turn off. Starting at 0x800, which should be the body or text of the functions of that executable, there is various data inserted and it is filled with data with another layer of compression.

[xatornet]

Has anyone been able to repack the file and use it? I'm doing some testing and the repacked file makes the game crash to desktop on loading.

I've just unpacked, compressed it and repacked it again. Just to make sure It works. No file modification done...

[OddOttAllen]

Has anyone been able to repack the file and use it? I'm doing some testing and the repacked file makes the game crash to desktop on loading.

I've just unpacked, compressed it and repacked it again. Just to make sure It works. No file modification done...

not as of right now, seems like repacking it makes the file bigger than it originally was for some reason... until we can figure out the cause, any mods affecting the game itself are on hold i think

[xatornet]

Has anyone been able to repack the file and use it? I'm doing some testing and the repacked file makes the game crash to desktop on loading. I've just unpacked, compressed it and repacked it again. Just to make sure It works. No file modification done...

not as of right now, seems like repacking it makes the file bigger than it originally was for some reason... until we can figure out the cause, any mods affecting the game itself are on hold i think

In my case, it actually compresses and repacks it in a size lower than the original file

[knight-ki11er]

Does anyone got a working image file with this? Or is this even possible?

[xatornet]

Does anyone got a working image file with this? Or is this even possible?

Not the game's image file. But you can change some of the emulator's settings, like there: https://github.com/Santikun/MGS1-Master-Collection-Bilinear-Patch

[kjetil-f]

What's the status on the NES and MSX games? Extractable?

[OddOttAllen]

What's the status on the NES and MSX games? Extractable?

NES games are 100% extractable through the bonus content app, it's built just like MGS1 with the alldata.bin file and everything... they're just .nes files but i haven't tested them in any emulators

MSX games is something that has yet to be figured out, i think someone has found that the MSX games are in .dll files? but that's all i'm aware of, i'm not sure how those would be extracted further

[kjetil-f]

Maybe the MSX games are ports and not emulated at all.

[aceofstax]

Maybe the MSX games are ports and not emulated at all.

Dunno, I'd think it'd be easier for the devs to just emulate. Thinking about the roms on SSBB, they were hidden in separate partitions of the Wii disc in their own main.dol files. Some of the files (GB/GBC? GBA?) on 3DS VC had the roms packed weird too. One of those VC platforms had the roms in the executable partition and not the romFS section IIRC. So there's possibility that people have either overlooked a partition or are just looking in the wrong spot.

FWIW, I own Master Collection for switch and it's packed similarly to how it's described in this thread. Contra Anniversary Collection was packed like the steam version too (same files, different media format and main binary). I also own multiple copies of the older MGS games.

I may be able to look into the different versions of MGS1 I own and compare them to the "washed" versions in Master Collection. I have the PSone books edition of MGS1 Integral (JP), Essential Collection MGS1 (US), Greatest Hits MGS1 (US) and PSN version of MGS1. Essential has the PS2 versions of MGS2 Substance & MGS3 Subsistence (non-HD). Subsistence features the MSX versions of MG1 & MG2.

I have not poked around yet, but I was crossing my fingers that I could eventually extract those MSX ROMs (if present) from the PS2 discs. If they're there, then that's another thing I can compare to Master Collection. HD collection MGS3 (with exception of MGS3D for 3DS)has MSX versions of MG1/MG2 playable too.

EDIT: Seems I found what "washed" is referring to. Compared PSClassic version of MGS1 disc 1 to MC MGS1 disc 1, and the only file with a different checksum is the SLUS005.94 file.

If you open this SLUS_005.94 file (used as a game ID) in a hex editor and replace it with all zeroes, you'll have "washed" the file the same way the Master Collection SLUS_005.94 file is. The master collection SLUS files are completely zeroed out. This explains why petmac had trouble loading the game in an emulator. The non washed SLUS file in hex editor has some SCEA text at the beginning of the file. Most emulators likely depend on the non-washed versions to load properly. But the rest of the disc (majority of the data) is identical to the PSClassic disc 1.

[ImanCol]

Muchos dicen que la versión de MSX es un port, pero si es igual a Ps2, primero miraría esa versión y ver si en internet alguien ya ha logrado ejecutar otros juegos para ese sistema o si o si es un port y no emulación.

Obtener Outlook para Androidhttps://aka.ms/AAb9ysg

---

From: aceofstax @.> Sent: Saturday, December 2, 2023 3:42:27 AM To: farmerbb/RED-Project @.> Cc: Marlon rodriguez @.>; Comment @.> Subject: Re: [farmerbb/RED-Project] Metal Gear Solid Master Collection (Issue #122)

Maybe the MSX games are ports and not emulated at all.

Dunno, I'd think it'd be easier for the devs to just emulate. Thinking about the roms on SSBB, they were hidden in separate partitions of the Wii disc in their own main.dol files. Some of the files (GB/GBC? GBA?) on 3DS VC had the roms packed weird too. One of those VC platforms had the roms in the executable partition and not the romFS section IIRC. So there's possibility that people have either overlooked a partition or are just looking in the wrong spot.

FWIW, I own Master Collection for switch and it's packed similarly to how it's described in this thread. Contra Anniversary Collection was packed like the steam version too (same files, different media format and main binary). I also own multiple copies of the older MGS games.

I may be able to look into the different versions of MGS1 I own and compare them to the "washed" versions in Master Collection. I have the PSone books edition of MGS1 Integral (JP), Essential Collection MGS1 (US), Greatest Hits MGS1 (US) and PSN version of MGS1. Essential has the PS2 versions of MGS2 Substance & MGS3 Subsistence (non-HD). Subsistence features the MSX ROMs of MG1 & MG2.

I have not poked around yet, but I was crossing my fingers that I could eventually extract those from the PS2 discs. If they're there, then that's another thing I can compare to Master Collection.

— Reply to this email directly, view it on GitHubhttps://github.com/farmerbb/RED-Project/issues/122#issuecomment-1837089968, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGHU5WIQ76CDJXTYWBNMMPLYHLSXHAVCNFSM6AAAAAA6OFKYCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGA4DSOJWHA. You are receiving this because you commented.Message ID: @.***>

[Mysterypome]

What's the status on the NES and MSX games? Extractable?

NES games are 100% extractable through the bonus content app, it's built just like MGS1 with the alldata.bin file and everything... they're just .nes files but i haven't tested them in any emulators

MSX games is something that has yet to be figured out, i think someone has found that the MSX games are in .dll files? but that's all i'm aware of, i'm not sure how those would be extracted further

Yeah about the bonus content, did they include that as well in the main product or were they absolute bastards and sealed it behind the pre-order?

[Mysterypome]

like advertised.

[Mysterypome]

My bad, I think, from what I'm seeing the bonus content does seem to be included with the $60 package.

[aceofstax]

@Mysterypome: Yes, it's included. Although it's unfortunately extremely likely to be a port. PS2 was a port. But with handhelds like Retroid Pocket 2S supporting simple PS2 games, maybe that's not a total bummer. Hopefully the ports in Master Collection will not require RP2S tier hardware for emulating these retro classics. It'd be nice to load them up on a pi zero or something.

Anyway...

Did more digging on what "washed" practically meant for each of the disc images included with this game. Again, this is the switch version I'm examining.

"washed" (as far as I can tell) simply means one or two (or both!) of the following:

• The SLUS (US) SLPM (JP) and SLES (EU) disc identifier files for each respective disc is zeroed out. Open the .bin image as an archive (7-zip; may need plugins to view) and you'll find the ID file either in the root of the image or in the "MGS" folder.
• The MGS1.EXE (or MGS2.EXE on disc 2) windows binary is zeroed out

The Switch release has an alldata.bin, a dlc_japan.bin, and a dlc_europe.bin. Only the dlc_europe.bin images featrue the zeroed out windows binaries ("Integral" releases for windows, I assume), but all of them have SLES ID files. If it's on the disc image and it's an .exe or an ID file, it's zeroed out. That's what "washed" means. The rest of the files seem to be untouched and original.

The following is true for the US release of MGS MC Vol 1:

US release disc images from alldata.bin (update 1.2.1)
SLUS_005.94 = MGS_US_DISC1-washed.bin
SLUS_007.76 = MGS_US_DISC2-washed.bin
SLUS_009.57 = mgs-vr-missions-ripped-washed.bin

JP disc images from dlc_japan.bin
SLPM_861.11 = mgs-disc1-ripped-washed.bin
SLPM_861.12 = mgs-disc2-ripped-washed.bin
SLPM_862.47 = INTEGRAL_DISC1-washed.bin
SLPM_862.48 = INTEGRAL_DISC2-washed.bin
SLPM_862.49 = INTEGRAL_DISC3-washed.bin

Non-Japanese disc images (.exe files... MGS Integral?) from dlc_europe.bin
SLES_013.70 = MGS_EN_DISC1-washed.BIN
SLES_113.70 = MGS_EN_DISC2-washed.BIN
SLES_015.06 = MGS_FR_DISK1-washed.BIN
SLES_115.06 = MGS_FR_DISK2-washed.BIN
SLES_015.07 = MGS_DE_DISC1-washed.BIN
SLES_115.07 = MGS_DE_DISC2-washed.BIN
SLES_015.08 = MGS_IT_DISC1-washed.BIN
SLES_115.08 = MGS_IT_DISC2-washed.BIN
SLES_017.34 = MGS_ES_DISC1-washed.BIN
SLES_117.34 = MGS_ES_DISC2-washed.BIN
SLES_021.36 = MGS_SPECIAL_MISSIONS-washed.BIN (no .exe on this disc)

Note that the only disc images with a zeroed out windows binary are the EN/FR/DE/IT/ES images. Every other image in the above list (including SLES_021.36) does not have a windows binary whatsoever. Not even a zeroed out one.

ALL of the above SLUS/SLPM/SLES ID files are zeroed out.

Did testing on ePSXe with both .bin and .iso copies of originals I own (MGS1 US, JP MGS Integral) and the washed images of the same release, only the originals with an intact SLUS/SLPM ID file booted up properly.

Worth mentioning that I don't believe just any ID file can replace the game specific ones... they make reference to files on disc, as well as the other disc in the release (example: SLUS_00594 references itself (disc 1) as well as SLUS_00776 (disc 2)). There's any chance of a transplanted ID file working, I think you'd have to hex edit the IDs of the donor title out and sub in the expected values. Even then there's the chance that memory card support could fail, tho save states in emulators could be a workaround for this specifically. Overall, seems it may actually need the original ID file to function as expected.

Personal opinion? Whether you have the originals or the washed disc images, I'd consider it ownership of the psx version. I just am not advising anyone implicitly or explicitly. It's possible there could be unwashed ID files hidden in another file such as the lz4a archives, but I have no idea where to start with those.

[Mysterypome]

I thought it was steam my bad.

[aceofstax]

i have an idea as to why this is.

it has to do with the BIOS verifying it as a PS1 game. i checked with some friends and it was a bit similar to the Ecco situation... we could see by comparing the game to a retail copy that a lot of the stuff the BIOS uses to verify if the game is real or not was taken out and put onto a separate file- we're hypothesizing for now that it's the .lz4a files. therefore, hanging at the PS logo.

i'm not an expert in this by any means so feel free to correct me, but i think the collection uses this method to skip a PS1 BIOS boot up? and since the BIOS on hardware/in emulators can't see this without it being put back in, it can't get into the game

I didn't see this before my above post, but I came to a similar conclusion. I'm not an expert either, just a tech and mgs enthusiast. I agree on both what you said about the bios stuff and the lz4a files.

I loaded my physical copy of MGS Integral (JP) and MGS (US) into HxD and compared them side by side. Use F6 to skip to next difference(s) enough and eventually you'll see why I don't believe transplanting a donor ID file will make them run like originals. Memory card handling seems to call functions with different names. All of the files outside of the ID file that exist on integral also exist on MGS1 (US) though. This is why I think a transplanted file might work, but it would definitely be better if the .lz4a archives contain the originals.

@Mysterypome No, you're fine. I think I'm the only one investigating this on the Switch edition in this thread so far as it's the only version I own. All posts outside of mine above are referencing the Steam release.

The two versions just have a lot of similar files that can be extracted the same way. I too had to use MArchiveBatchTool to find the disc images. The only file I needed @hadess' fork of MArchiveBatchTool for was the dlc_europe.bin (about 7-8GB) in the Switch release.

[Mysterypome]

I'll be honest when it come to dealing with collections I took no risks with the switch.

[Mysterypome]

I'm still trying to figure out how to extract the arcade roms from the switch.