BUG: mkpsxiso v1.06 crashes if any directory contains more than 32 files

[MeganGrass]

If any given directory/folder contains more than 32 files, mkpsxiso will crash just as it appears to be finalizing the disc.

As a result, the cue sheet is not generated and the BIN image appears to be useless.

[Lameguy64]

Looks like the crash was caused by an overflowing directory record buffer as the program writes down the ISO9660 file system due to too many file/directory entries specified in a single directory. It appears that the check I've added wasn't so effective so I'm going to have to add a more effective check by making the program error out when there are more than 30 entries specified in a single directory just like BUILDCD as the PlayStation cannot parse directories containing more than 30 entries due to library limitations (assuming you're using the official PlayStation SDK, not sure how PSXSDK parses the ISO9660 file system so I don't know the max number of files it can parse).

If you want to store more than 30 file entries in a single directory, you're better off using a custom pack file system which I've actually written a library of that I've use for my own projects (but its only for the official PlayStation SDK). I can uploaded it as a new git repo if you want it and that I feel it would come in very handy for other PlayStation homebrew developers as well.

On Wed, Jan 25, 2017 at 10:04 AM, MeganGrass notifications@github.com wrote:

If any given directory/folder contains more than 32 files, mkpsxiso will crash just as it appears to be finalizing the disc.

As a result, the cue sheet is not generated and the BIN image appears to be useless.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Lameguy64/mkpsxiso/issues/2, or mute the thread https://github.com/notifications/unsubscribe-auth/AJxt7-4NWbuPx4qA4jqYiUIYw_6PhVI-ks5rVq3JgaJpZM4LtBMK .

[MeganGrass]

Hmmm.... well, the current project that I'm working on is actually a modification to an already existing game - Resident Evil 2 Dual Shock. To be more specific, I am simply modifying the original executable with armips and recompling the disc image.

There is a folder titled "BSS" on the root directory which contains 127 files, by default. This is not a modification - the game is originally built that way.

Sony's BUILDCD application, dated Thursday, ‎August ‎20, ‎1998, ‏‎8:49:10 PM, has absolutely no problem whatsoever compiling an image using the default layout of that specific game. Since I no longer have an x86 operating system, I am now dependent on mkpsxiso (which is utterly fantastic, btw).

I do believe that the library functionality that you speak of does contain the specific limitations that are mentioned.

-However-

CAPCOM never strictly adhered to those specific library functions and instead programmed their PSone games to load data by referencing an internal LBA table within the executable, via functionality like "DsIntToPos()", "DsSyncCallback()", "DsCommand()", etc. Doing so allowed them to drastically cut loading time by bypassing the bloated PSYQ functionality. It is also my understanding that many non-CAPCOM games implemented similar methods that do the exact same thing.

[Lameguy64]

Ah, I see. Luckily, I already have an iso image of RE2 and based on what I've analyzed with directory entries that exceed a 2048 byte sector, its just a matter of writing directory records across multiple sectors which should be pretty easy to implement.

Yeah, as a PlayStation homebrew programmer myself, one way to bypass the 30 file per directory limit of the CD libraries would be to reference files directly by LBA rather than referencing a file through the ISO file system. The reason why file access times are slow if you reference files via file system is not because of the 'bloat' of the CD libraries but rather the amount of time it would take for the CD library to read through the file system and then having to seek to the file you want to read.

I actually found ways of getting around the slow file search issues and the 30 file per directory limit by making my own pack-file system that only required a single file system search just for locating the pack-file itself and the rest has all the LBA information of the files stored in memory and therefore, speed up file access times considerably without having to hard-code LBA addresses to my program. Much more so when I pack a bunch of small files such as textures and sound effects into a single file which are loaded all at once with just a single read command. Such optimization tricks made my Marilyn homebrew game feel almost like a cartridge based game with no loading screens at all.

Anyhoo, I'm off to work on the update.

On Wed, Jan 25, 2017 at 11:43 AM, MeganGrass notifications@github.com wrote:

Hmmm.... well, the current project that I'm working on is actually a modification to an already existing game - Resident Evil 2 Dual Shock. To be more specific, I am simply modifying the original executable with armips and recompling the disc image.

There is a folder titled "BSS" on the root directory which contains 127 files, by default. This is not a modification - the game is originally built that way.

Sony's BUILDCD application, dated Thursday, ‎August ‎20, ‎1998, ‏‎8:49:10 PM, has absolutely no problem whatsoever compiling an image using the default layout of that specific game. Since I no longer have an x86 operating system, I am now dependent on mkpsxiso (which is utterly fantastic, btw).

I do believe that the library functionality that you speak of does contain the specific limitations that are mentioned.

-However-

CAPCOM never strictly adhered to those specific library functions and instead programmed their PSone games to load data by referencing an internal LBA table within the executable, via functionality like "DsIntToPos()", "DsSyncCallback()", "DsCommand()", etc. Doing so allowed them to drastically cut loading time by bypassing the bloated PSYQ functionality. It is also my understanding that many non-CAPCOM games implemented similar methods that do the exact same thing.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Lameguy64/mkpsxiso/issues/2#issuecomment-275011257, or mute the thread https://github.com/notifications/unsubscribe-auth/AJxt782QxmcjynJsXpM6gLSudZqehr1Eks5rVsTZgaJpZM4LtBMK .

[Lameguy64]

Okay, I've quickly put together this 1.07test release which should be able to create a directory with more than 32 entries without crashing: http://lameguy64.github.io/mkpsxiso/mkpsxiso-1.07test.zip

Because I cannot test this new implementation myself extensively (I only tested it with around 45 directory entries) due to time constraints with personal stuff, I'm going to need it tested extensively with your project before I put it out as a main release.

[MeganGrass]

I too, have written my own functionality which utilize an internal LBA index and parses file containers accordingly :) When I mention bloat, well, I really mean that in the sense of bypassing all the standard SDK functionality to get right to the purpose - loading files. In my years of research, I've noticed that games which do not implement their own LBA-loading techniques take longer to load data simply because of all the functionality that must be processed before the data is actually loaded. Obviously, bypassing that will free up precious processing cycles and doesn't play hell with the laser, not to mention that the previously aforementioned restrictions can be bypassed.

Okay, onto the subject at hand... I have tested the build that you have provided and it was able to properly compile the disc image, however, something seems to have gone wonky with the LBA.

Here is a partial build log provided by BUILDCD:

Lpath              00:24:18  1        18        
OptLpath           00:24:19  1        19        
Mpath              00:24:20  1        20        
OptMpath           00:24:21  1        21        
Dir()              00:24:22  1        22        
Dir(BINARY)        00:24:23  1        23        
File(BIO1.BIN)     00:24:24  8        24        00:24:23   07/20/1998 22:43:42   16384      Form1 C:\BIODS\DISC\BINARY\BIO1.BIN
File(BIO2.BIN)     00:24:32  8        32        00:24:23   07/20/1998 22:43:42   16384      Form1 C:\BIODS\DISC\BINARY\BIO2.BIN
File(BIO2P.BIN)    00:24:40  8        40        00:24:23   07/20/1998 22:43:42   16384      Form1 C:\BIODS\DISC\BINARY\BIO2P.BIN
File(BIO3.BIN)     00:24:48  8        48        00:24:23   07/20/1998 22:43:42   16384      Form1 C:\BIODS\DISC\BINARY\BIO3.BIN
File(BOOT.BIN)     00:24:56  12       56        00:24:23   07/20/1998 22:43:42   24576      Form1 C:\BIODS\DISC\BINARY\BOOT.BIN
File(CONFIG.BIN)   00:24:68  14       68        00:24:23   07/20/1998 22:43:42   27756      Form1 C:\BIODS\DISC\BINARY\CONFIG.BIN
File(DEBUG.BIN)    00:25:07  7        82        00:24:23   07/20/1998 22:43:42   14336      Form1 C:\BIODS\DISC\BINARY\DEBUG.BIN
File(DIEDEMO.BIN)  00:25:14  8        89        00:24:23   07/20/1998 22:43:42   14536      Form1 C:\BIODS\DISC\BINARY\DIEDEMO.BIN
File(ENDING.BIN)   00:25:22  24       97        00:24:23   07/20/1998 22:43:42   49152      Form1 C:\BIODS\DISC\BINARY\ENDING.BIN
File(MEM_CARD.BIN) 00:25:46  12       121       00:24:23   07/20/1998 22:43:42   23932      Form1 C:\BIODS\DISC\BINARY\MEM_CARD.BIN
File(OPENING.BIN)  00:25:58  7        133       00:24:23   07/20/1998 22:43:42   14336      Form1 C:\BIODS\DISC\BINARY\OPENING.BIN
File(RESULT.BIN)   00:25:65  9        140       00:24:23   07/20/1998 22:43:42   17360      Form1 C:\BIODS\DISC\BINARY\RESULT.BIN
File(SELECT.BIN)   00:25:74  10       149       00:24:23   07/20/1998 22:43:42   18760      Form1 C:\BIODS\DISC\BINARY\SELECT.BIN
Dir(BSS)           00:26:09  4        159       
File(BH1TIT.TIM)   00:26:13  76       163       00:26:09   07/20/1998 22:43:42   153620     Form1 C:\BIODS\DISC\BSS\BH1TIT.TIM

....and here is a partial build log provided by mkpsxiso v1.07 test: BIN LBA:23 +-BIO1.BIN;1 LBA:24 +-BIO2.BIN;1 LBA:32 +-BIO2P.BIN;1 LBA:40 +-BIO3.BIN;1 LBA:48 +-BOOT.BIN;1 LBA:56 +-CONFIG.BIN;1 LBA:68 +-DEBUG.BIN;1 LBA:82 +-DIEDEMO.BIN;1 LBA:89 +-ENDING.BIN;1 LBA:97 +-MEM_CARD.BIN;1 LBA:121 +-OPENING.BIN;1 LBA:133 +-RESULT.BIN;1 LBA:140 +-SELECT.BIN;1 LBA:149 BSS LBA:159 +-BH1TIT.TIM;1 LBA:255

NOTE: Everything seems to align perfectly, until a folder is reached/parsed and mkpsxiso somehow manages to go wrong. Specifically, notice that the "BSS\BH1TIT.TIM" file is supposed to be located at LBA sector 163, whereas mkpsxiso pushes it all the way back to 255.

On a side note, yes, this disc structure is obviously greatly different compared to the original RE2 ISO hahaha Remember, I did mention that I created my own file containers and such! A few simple modifications to the original executable here and there and voila. All of this is regardless to the matter at hand, however. Compiling with BUILDCD has never been a problem; I only didn't want to confuse you on that!

[Lameguy64]

Ah, looks like the new LBA calculation method turned out to be a bit screwy as it calculates file LBAs a few sectors ahead of the directory record entry which also results in completely unwritten sectors that have no subheader information which wouldn't be good as CD drives needs it for tracking sectors during seeks.

Here's a new test build that fixes the LBA calculation bug: http://lameguy64.github.io/mkpsxiso/mkpsxiso-1.08-test.zip

[Lameguy64]

So... Problem solved and working great?

[MeganGrass]

Forgive me for the much delayed response, as I haven't had any time to work with this at all.

I'm still experiencing a similar error, though it's not nearly as bad as before. Specifically, the LBA is still slightly off when a folder is encountered:

+-SELECT.BIN;1 LBA:149 BSS LBA:159 +-BH1TIT.TIM;1 LBA:161

The "BH1TIT.TIM" file should have an LBA of 163.

We're getting close!

[MeganGrass]

In the "int iso::DirTreeClass::CalculateTreeLBA(int lba)" function, I've noticed this method will get the correct LBA:

// Set LBA of directory record of this class
recordLBA = lba;

// lba += (CalculateDirEntryLen(true)+2047)/2048;
int _Size = (numentries * 64);
int SectorSize = (_Size + 2048 - 1) / 2048 * 2048;
SectorSize /= 2048;
lba = (recordLBA + SectorSize);

I compiled the app with Code Blocks and yet the program is still buggy, outputting a corrupted ISO albeit with proper LBA entries. Playstation emulators refuse to boot it and ISO viewing apps such as CDmage spit out warnings about a corrupt file system.

[Lameguy64]

Late response as I've been pretty busy on a bunch of things as of late, sorry about that.

I'm going to have to take a look at it more in-depth when I have the time. I think certain file directory records had to be padded a bit more by the looks of it.

Can you send me your test xml project and files?

[MeganGrass]

No worries, I totally understand :)

Here ya go: https://drive.google.com/open?id=0B5megDVWOileRmJxVWJKSDZGV2s

Thank you for your time and patience with me!!

[Lameguy64]

Okay, it looks like I have finally fixed the LBA bug! Turns out my LBA calculation routines we're a bit bugged which led to files being assigned inside a directory record that exceeds more than one record which then get overwritten by the directory record leading to data corruption.

Attached is a fresh new test build of mkpsxiso which contains the above fix as well as a few other bugfixes: mkpsxiso-(1.14-test).zip

According to my tests, the LBA assignments are now more or less 1:1 to the buildcd logfile you provided. Your issue with mkpsxiso should be solved with this shiny new test build.

[MeganGrass]

Excellent, thank you so very much! @Lameguy64 :1st_place_medal:

I've noticed that ISO applications still throw a fit when opening the output, but the game finally boots and plays fine now. The sizeof the output file doesn't match what BUILDCD creates, but regardless. It works now and I am very thankful for that! :)

[Lameguy64]

You're welcome! I'm glad to hear that it is finally fixed. I'm still going to do some additional tweaking before pushing out an update.

What ISO programs were you using that threw a fit when you opened the images my tool generated though?

[MeganGrass]

CDmage, specifically:

[Lameguy64]

Well, CDmage still throws a fit when opening images generated with the latest release of MKPSXISO but it appears that it still works okay and displays all the files in the list despite the error. Looks like the ISO generation routines sill needs a bit more tweaking and will require analyzing ISO images further but for now, I need to do something else and I'll get to that in the future.

[Lameguy64]

Just released version 1.15 of MKPSXISO which should fix issues like that when opening generated images with ISO tools. I've also done some tweaks to the file system encoding and at this point, ISOs generated by MKPSXISO should more or less be 100% ISO9660 compliant.

[MeganGrass]

Excellent job well done, thank you :)