search

ps2homebrew / Open-PS2-Loader

Game and app loader for Sony PlayStation 2
https://ps2homebrew.github.io/Open-PS2-Loader/
Academic Free License v3.0
2.05k stars 262 forks source link

Caching game titles - when there are many games installed #158

Closed sp193 closed 5 years ago

sp193 commented 5 years ago

I'm reluctant to do this because I feel like it's more of a want than a need, but because people do this anyway...

OPL seems to have issues with extremely long loading times at boot, when there are many games installed. This happens when:

For the HDD unit, the cause is the design of APA, which is the partitioning scheme that we use on the PS2. For forward and backward-compatibility with Sony software, as well as other software. Due to the design of the APA driver and the IOP only having 2MB, it becomes impractical to maintain a large cache. By hard-coded limits, it will not maintain accesses to more than 128 or so partitions, which most disks larger than 128GB will likely easily exceed. How it cached the existence of partitions, is by recording the whole APA header in memory, which is 1KB per partition. This is why the HDD Browser becomes very slow when it is made to access a disk with sufficient partitions on it; when it tries to access partitions beyond the limits of the cache, it will start to crawl due to APA perpetually evicting the next partition it needs to access from the cache, if dread() is mixed with a call to open(). There is no partition table in this format, meaning that all partitions have to be scanned through, which results in long seek times.

OPL avoids this by just accessing the game title information directly by reading the sectors with devctl(), but it seems to still take a while to list the hundreds of partitions on a sizable disk (perhaps 400~800 partitions on a 2TB disk).

An obvious solution would be to record all the partitions that represent games in a cache, but the existing format does not have such a facility. So whenever the user changes the games on the HDD with a tool that is not aware of this cache, will cause the cache to go out of date.

There's also this possibility of changing the format, with one that has support for a centralized game list. By changing the game list, it will also become possible to get rid of a horrible problem that the HDLoader game format has - it is likely incompatible with the HDD Utility Disc's optimization option, as that could rearrange the partitions on the disk without updating the HDLoader game structure of the game, hence ruining the game. However, changing the game format will also require users to move forward to the new format (with no possibility of moving back), which I don't know if anybody is interested in that....

For USB devices & SMB: USB is generally slow, but I think the actual problem might come from users who do not name their ISO images in the old format: SXXX_YYY.ZZ.Game Name.iso This would cause OPL to mount the ISO and parse SYSTEM.CNF, to obtain the boot filename, which used to be specified within the filename of the file (SXXX_YYY.ZZ).

If the performance of iterating through (without mounting) 400 game titles is actually acceptable, then it probably becomes quite easy to automatically maintain a games list: it just has to iterate through all ISO files to identify what has changed, before updating the game list. But, I still have no idea whether this is really the case or not.

Badore90 commented 5 years ago

...An obvious solution would be to record all the partitions that represent games in a cache, but the existing format does not have such a facility. So whenever the user changes the games on the HDD with a tool that is not aware of this cache, will cause the cache to go out of date.

...However, changing the game format will also require users to move forward to the new format (with no possibility of moving back), which I don't know if anybody is interested in that....>

It would be possibile to not do such drastical change??

I mean, there's no problem about the tools we already use (for install/extract/edit games…), it should be an OPL only related feature.

For people that aren't interested, Jay-Jay thought of a solution: http://www.ps2-home.com/forum/download/file.php?id=11288&mode=view this option will be OFF by default.

Therefore it would be simply like that:

sp193 commented 5 years ago

That is the current plan, but because there is no way to indicate to the user that the game cache is out of sync, it is proabbly quite a breaking thing to the user experience. And if I were to re-enable the refresh button, it could also be disruptive to users who choose to have auto-refresh enabled (whenever OPL refreshes all game lists, it'll take ~30-40s because of the HDD). I could also make it so that the button only works once per boot... but that is perhaps quite unintuitive. Other than trying to get more complete information on this problem, those were also the reasons as to why I am bringing this up here.

And if we choose to make it so that OPL can maintain the game list of ISO files on USB devices and SMB shares, then it would probably be more preferable to have this function always enabled.

So other than making a new feature that does as intended, it should also have an actual improvement of the user's experience.

Badore90 commented 5 years ago

That is the current plan, but because there is no way to indicate to the user that the game cache is out of sync, it is proabbly quite a breaking thing to the user experience.

When I add a game I'm aware of it… If I'm drunk I'll just press the refresh button, being in doubt! (well actually it never happen, I've crystal clear ideas when I'm drunk...).

It wouldn't be difficult to spread the voice among huge game-list owners (I'm pretty sure they'll notice the new option immediately, they're always quite experts, and if you find a newbie whit a huge list, sure someone expert made it, and tell to the newbie how it works), it would be a tangible improvement I think.

Will have OPL preserving the Ps2 immediacy, even with a huge iHDD gamelist. IMHO it's the only minus (among a lot of plus) comparing OPL to the "official ps2 gaming".

Badore90 commented 5 years ago

Plus, there's to say that who make an internal HDD huge list, don't even use the refresh button, 'cause usually you install your games all in a once. You simply fill the HDD with your list, then don't delete nothing.

haudi commented 5 years ago

Is that so simple? I never installed 128+ games on the HDD 'cause usually you delete a game... sooner or later. Just saying.

yoshi314 commented 5 years ago

i don't really have a problem with this on hdd. it was atrociously slow on HDloader (several minutes), whenever you would install/remove a game from the disk. So OPL is fast enough here.

When it comes to USB - it would make sense. Never used SMB, so i cannot comment there.

That is the current plan, but because there is no way to indicate to the user that the game cache is out of sync, it is proabbly quite a breaking thing to the user experience.

Just refresh when game list is out of date on program startup. Also, give an option to manually refresh, when using SMB, since the game list might have changed server-side. Keep it simple.

Badore90 commented 5 years ago

I never installed 128+ games on the HDD 'cause usually you delete a game... sooner or later. Just saying.

The goal (of this feature) is just to not have to delete any game instead. It is an increasingly common custom to have something like this: https://youtu.be/4WJFQmoObMc?t=472

For me i.e. OPL takes 22 (26 at first boot) secs to load my list (about 400 games). Having this option instead it would be like having the game disc inserted, just power-on and play (even with 700 games on a 2tb HDD). I think it would be a tangible improvement for those with 6/700 games, that have to wait about 40 secs.

It's a good thing for testing purposes too (that is one of my reasons for making my pretty huge game-list). It's a little boring when testing compatibilities, having to wait 20/30 secs after every IGR...

haudi commented 5 years ago

I'm OK on USB and HDD but a new image format is interesting. Are there maybe more 'flaws' on the design, which can be fixed with a new format? I know, it's more of a want than need like sp193 said but (personally) for a embedded machine from 2000 it's an interesting aspect in terms of a more modern image format.

@Badore90 On that point testing games can be annoying. You are right. I just wanted to say it's not obvious that everyone has so many games installed :-)

sp193 commented 5 years ago

While it's good to let OPL manage the game list by letting it scan the device and update the list at boot, that is also the problem that we're trying to avoid for the HDD unit (it takes too long). I think only SMB and/or USB devices might not have this problem, but there is currently a lack of data on it. But I may implement something soon, to let us play around with.

As for other problems with the current HDLoader game format that I know of: Perhaps because the only filesystem that Sony allowed for use for games on the PS2 was PFS, their PSX XMB seemed to assume that the partitions use PFS (even though HDLoader games have some other format identifier). While such games appear as "legacy" titles under the XMB, bringing up the context menu to do anything (i.e. delete, view properties of) would crash the XMB. I don't know whether we should bother to address this, as I feel that few people have PSX consoles that work and they probably should not be used for everyday gaming either (their HDDs are dying)... But to address this, we would perhaps need to store the game in PFS, within APA. Rather than just within the APA partition. However, we will likely still be unable to take advantage of PFS because we try to go without any filesystem driver, for saving memory. Even though the XMB should probably be fixed instead, it was just made like that and PSX consoles come with it...

In an earlier post, I mentioned that using the HDD Utility Disc to optimize the disk could break HDLoader games. It's only a problem if one used the Sony HDD Utility Disc, but HDDChecker should be capable of optimizing the disk while keeping the games safe. So it is not like we cannot use this format safely today...

@yoshi314 I never knew that HDLoader was worse! Perhaps it also accessed the partitions through HDD.IRX. Thanks for sharing.

Badore90 commented 5 years ago

Yes HDLoader takes a little more than 40 secs to load my list (versus 22 secs of OPL). The real problem with HDLoader is when it build the gamelist (first time you access it, than every time you install/delete a game). For a 400 gamelist it takes one hour. It doesn't matter how much you change, even if you delete/add just one game, HDL will re-build the list from scratch.

carl0sjt commented 5 years ago

would be very helpful to show number of displayed items (and maybe "invalid" files...?). some times its hard to spot a .iso that OPL cant list...

sp193 commented 5 years ago

But if it cannot be listed, then it probably cannot be booted. There aren't any checks to determine whether the game is a valid game or not, other than some checks to ensure that the way that the user named the file will not crash OPL. And so, the game will have to be a .iso file. If the boot filename is not specified in its filename, the game must also have a valid SYSTEM.CNF file (all PS2 games do).

carl0sjt commented 5 years ago

I have an ISO here from State of Emergency game, ID is SLUS_202.14. It's named "State of Emergency (USA).iso", CRC32, SHA1 and size matches the redump.org site. It seemed to boot before, (I cant test now, cause my PC, at this moment is doing intensive tasks, and disk usage is 100%, so this causes OPL to not display any games. This spefic ISO, won't show on game list if the game ID is not present in the name of the file. I've checked file permissions on Windows, and file access/write is set to everyone. One thing that call my attention to this ISO is that the raw file size 4.34 GB, but with NTFS compression, the file occupies 503 MB of disk, so this ISO has a lot of "padding", "zeroes", IDK if NTFS might be causing this problem or its the ISO itself.

sp193 commented 5 years ago

It might be the SYSTEM.CNF parsing code of OPL, which is hence a bug that should be taken care of. If you can boot the game with the boot filename manually specified, then it is not a permissions problem and the disc image can be accessed properly.

The code for mounting the ISO was taken from OPL's CDVDMAN module, so it should be fine.

It is not unheard of for games to have large regions of padding, to keep the data in the outer tracks of the disc, to improve reading performance. But even if NTFS compression is enabled, software reading the files will not encounter a difference in file contents.

Badore90 commented 5 years ago 
      I have an ISO here from State of Emergency game, ID is SLUS_202.14. It's named "State of Emergency (USA).iso", CRC32, SHA1 and size matches the redump.org site. It seemed to boot before, (I cant test now, cause my PC, at this moment is doing intensive tasks, and disk usage is 100%, so this causes OPL to not display any games. This spefic ISO, won't show on game list if the game ID is not present in the name of the file. I've checked file permissions on Windows, and file access/write is set to everyone. One thing that call my attention to this ISO is that the raw file size 4.34 GB, but with NTFS compression, the file occupies 503 MB of disk, so this ISO has a lot of "padding", "zeroes", IDK if NTFS might be causing this problem or its the ISO itself.

State of Emergency is a strange one. I played it sometimes from internal HDD. It need MODE 2 enabled to not crash at the (second if I remember correctly) loading screen (I was using OPL 0.9.3 at that time).

However after IGR on this game, 90% of times OPL couldn't show the gamelist. It load just for one second then the HDD page showed blank. I always IGR to OPL and never had problems with any game, but I think to remember that even IGResetting to FMCB/OSDSYS with this game the problem remained, I needed to power-off the Ps2.

I thought it's related to the HDD cache. But this is the only game out of +400 with this problem.

sp193 commented 5 years ago

But has it happened recently? I have changed OPL a few months ago, to ensure that no ongoing transfers will be interrupted by IGR.

Badore90 commented 5 years ago

I'm using OPL_DB_1330 now. I'll try the game and make you know.

Another question: it's possible to add an option in OPL to power-off the iHDD??

Badore90 commented 5 years ago

Well, I can't make this game (State of Emergency) to work anymore, it freeze at the first loading screen. Anyway I can IGR and OPL always loads the gamelist correctly.

Anyway this game has something weird for sure. After booting it, the Ferrari challenge freeze rise again. (btw I can IGR even at the Ferrari Challenge freeze!! I Always had to hard-power-off the Ps2 after that. I don't know if it's due to OPL or to my new SATA board….).

I could make State of Emergency work last year, with MODE 2 and MDMA 0. Now it doesn't work in any way (even with OPL 0.9.3 on which I used to play this)…

Btw I think it's better to stop here the O.T. '^^

sp193 commented 5 years ago

And... we have something: http://www.psx-place.com/threads/opl-with-prototype-game-list-cache.22036/

For USB & SMB shares, the list is automatically maintained by OPL and it will cache ISO files that do not have the boot filename in their names (i.e. mygametitle.iso).

For the HDD unit, the game cache is used by default and will only be refreshed if you press the refresh button. Automatic refresh of the HDD unit is disabled because it does not make much sense; the HDD cannot be hot-plugged.

If you have a large number of games, please consider helping us by trying this out and letting us know how the user experience is like. There is nearly no risk, given that the game list cache is a new file.

Testing on my end has been minimal so far.

Download: https://www.sendspace.com/file/h1wjyl Github branch: https://github.com/sp193/Open-PS2-Loader/tree/gamelistcache

sp193 commented 5 years ago

I realized that there were problems with ETH and HDD shutting down DEV9, caused by their deinit() functions being always called by the UI regardless of whether these modes were used or not.

Badore90 commented 5 years ago

Thank you very much!!

One question: I've the OPL (with all OPL cfg files inside) folder on my MC. On the iHDD I've Ps2 games and other 3 partitions only (mbr, HDLoader settings and OPL)

Where the cache-gamelist file will be saved??

sp193 commented 5 years ago

+OPL partition, on the HDD. As with the game settings.

HDD: hdd:/+OPL/games.bin Others: CD/games.bin & DVD/games.bin

sp193 commented 5 years ago

I realized that it might not always update the game list correctly, so please use this new file: https://www.sendspace.com/file/h1wjyl

sp193 commented 5 years ago

New changes for today:

For the new APPS feature, applications can be put within their own folders, within the APPS folders on each supported device. For example, on a USB disk:

APPS/
    LaunchELF/
        title.cfg
        BOOT.ELF
    SMS/
        title.cfg
        SMS.ELF

title.cfg only requires two lines for now. For example:

title=LaunchELF v4.50
boot=BOOT.ELF

Download: https://www.sendspace.com/file/vpnnae EDIT: please use the new file.