[Feature Request]: split balika's MagicGate commits

AKuHAK commented 2 years ago

Description

balika's MagicGate commits are mixed with Python2-specific commits, it will be better to split them into different commits as it's done, for example, here.

Also, is it worth asking if you have an NVRAM file collection for this specific architecture?

Reason

.

Examples

.

987123879113 commented 2 years ago

The history got wiped when I was trying to redo master to make maintenance easier on me a few weeks ago (had lots of unnecessary merge conflicts and such). I do have a backup of the old master branch which I pushed now if you wanted something in particular out of it: https://github.com/987123879113/pcsx2/tree/master_old

There are some other commits that I didn't want to wipe out as well so I will probably go through and try to fix that all up once I finish the PCSX2 Qt rewrite.

Also, is it worth asking if you have an NVRAM file collection for this specific architecture?

I have dumps from various machines but this particular hardware is just a stock SCPH-50000 so I don't know if there would be much of interest.

987123879113 commented 2 years ago

Also, just to be clear in case you were worried about the credit aspect of the commits: they're credited on the Wiki page since I used Florin9doi's branch as a base when pulling in the MG PR. It originally was credited in the README.md but any time upstream changed README.md, which is/was often, I'd have to manually update the branch (instead of using Github's sync with upstream button which is what I want to use most of the time) so I just moved all branch-specific things to the Wiki page.

https://github.com/987123879113/pcsx2/wiki

AKuHAK commented 2 years ago

I have dumps from various machines but this particular hardware is just a stock SCPH-50000 so I don't know if there would be much of interest.

It is not completely true, cause in reality stock 50000 just does not exist. There exist MB/NH, NB, TSS, CW, SA, and PW versions, but not stock non-colored 50000. For MechaPwn research purposes, I am collecting all possible NVRAM variations and it looks like that pure simple black 50000 was only used inside Python2 machines. So if you can provide NVRAM backup with sticker photo for any machine (and with mechacon version if possible), it will help a lot in research. It is also possible that Konami machines are using the mechacon patch area inside nvram for something interesting.

I do have a backup of the old master branch which I pushed now if you wanted something in particular out of it: https://github.com/987123879113/pcsx2/tree/master_old

oh, thanks. I am not so worried about credits, but about the ability to merge your work

987123879113 commented 2 years ago

I am collecting all possible NVRAM variations and it looks like that pure simple black 50000 was only used inside Python2 machines.

I've never heard of a black SCPH-50000 in a Python 2 before. It's always been the MB/NH variant. I can send some dumps your way if you can provide an email address or some way to send it over that doesn't involve directly attaching here (original hardware owners are on the paranoid side). Edit: Or send an email to 63495610+987123879113@users.noreply.github.com saying it's you and I'll respond with some dumps. Edit 2: Finished meeting and re-read your message again. I don't have physical access to the hardware since it's stuff a collector friend dumped for me so I can't provide any pictures of labels unfortunately. The most I can provide is the dumps I have on hand already, which consist of the BIOS, EROM, NVM, and ROM1/ROM2 files. I can put you in contact with another friend who has physical hardware on hand that could probably also provide you with more things including pics if required.

987123879113 commented 2 years ago

Email sent. Feel free to edit out or delete the message with your email address.

AKuHAK commented 2 years ago

I checked NVRAM backups, and found so far:

you were correct - that is just regular SCPH-50000 MB/NH, model ID matches
it looks like this later model, not the launch revisions, but the one that was released when the first slims were launched
all of them are J chassis and plant SZMT (this should be visible on the sticker: J SZMT)
J chassis means that all of them have 190J Bootrom and 3.03J DVDrom
3 of them were shipped with 5.12 mechacon and one with some 5.0, 5.2 or 5.4 mechacon So for better future pcsx2 support you can modify *.mec file (that is generated along with bios files in the pcsx2 bios folder) and type these 4 hex bytes inside: 00 05 0c 00. Currently, pcsx2 doesn't emulate mechacon fully, but if it will be better to keep the correct mechacon version.

987123879113 commented 2 years ago

@AKuHAK Thanks for the info. You mentioned 3.03J for the DVD ROM which made me remember another set of dumps from a friend I received recently. In particular, the machine he had 3.02 which was causing us some issues since we were attempting to make some custom discs and wanted to use FreeDVDBoot. I sent the additional dump + the other info I have on the machine to the same email address.

AKuHAK commented 2 years ago

This machine is the only one that is not J chassis, but I chassis (there should be on sticker I SZMT). Unfortunately, FreeDVDBoot currently doesn't support <3.03 DVD player, so you can use it only for J chassis not for I chassis.

Basically, I SZMT on the sticker will not support FeeeDVDBoot, J SZMT will support FreeDVDBoot.

For such sort of questions, you are kindly welcome to official Discord: FreeDVDBoot creator also available there.

987123879113 commented 2 years ago

I redid master using Florin9doi's latest magicgate branch as a base and squashed old master on top and confirmed things are still working, so I'm going to close this issue now.

Qt rewrite still isn't in master so if you were planning on using the code for something then I suggest maybe holding off for a few weeks until the Qt rewrite changes + additional bugfixes hit master and things settle again.

AKuHAK commented 2 years ago

@987123879113 just would like to mention that I enabled Codacy warnings for your work. In case you will found that interesting, you can take a look here: https://app.codacy.com/gh/ps2homebrew/pcsx2/pullRequest?prid=9877736

987123879113 commented 2 years ago

Thanks, I fixed up some things mentioned in the warnings. Anything else I didn't fix is stuff I don't intend on fixing (warnings for the old wxWidgets GUI code notably).

AKuHAK commented 2 years ago

@987123879113 can you merge latest changes from magicgate fork? Its not related to Python2 functionality, I just dont want to keep separate version for testing DVD players on ps2 :)

987123879113 commented 2 years ago

Done. I tried to fix some merge conflicts so hopefully next time I don't need to manually fix things. Florin9doi keeps rebasing the branch and every time I have to manually fix it, so if I have to manually fix it every time then I probably won't update the code anymore unless something significant changes.

Edit: The gamedb linter is also broken now with the addition of a new field, so I had to manually fix that as well. Please make sure that stuff is working too before requesting it to be updated again. MacOS builds are now broken too it looks.

AKuHAK commented 1 year ago

As Discussions in this fork are disabled, ill post my investigations here: Some action to make Namco System 246/256/ Konami Python 1 work on pcsx2: As MagicGate PR doesn't add support for mcman and secrman for arcades, some preparations need to be done. Namco System 246 (0100TZ20000306)(20021119-163841) bios needs to be modified: MCMAN and MCMANO (I suppose that O stands for Original) should be swapped (I just changed their names in the module list): SECRMAN replaced with retail version extracted from bios that is used for Python2. As the retail secrman module is smaller, I just put it over without changing its size in the module list. Offset 0xC5540.

Now as the bios is prepared, time for dongles. I was playing with sed1ver as an example. Below is a python script for creating proper memory card images for pcsx2:

import sys
import os

block = 512
crc = 16
input = open(sys.argv[1], "rb")
input_ecc = open(os.path.splitext(sys.argv[1])[0] + "_spr.ic002", "rb")
out = open(os.path.splitext(sys.argv[1])[0] + ".ps2", "wb")
while True:
    data = input.read(block)
    out.write(data)
    if not data:
        break
    ecc = input_ecc.read(crc)
    out.write(ecc)
    if not ecc:
        break
out.close()
input.close()
input_ecc.close()

This script will combine sed1vera.ic002 and sed1vera_spr.ic002 into sed1vera.ps2 for use in pcsx2.

All arcade dongles use mc0:boot.bin as an entry point. As we changed keys when bios was modified, boot.bin will fail to decrypt. So we need to prepare it.

For extracting I use mymc app:

mymc sed1vera.ps2 extract boot.bin

For decrypting - the latest kelftool. Don't forget to put PS2KEYS.dat for arcade in the program folder.

./kelftool decrypt boot.bin boot.elf

So now it is time to configure pcsx2. Select modified bios, insert the modified memory card into the empty slot, sed1dvd0.iso as ISO source. Now just run decrypted boot.elf (as I don't know how to run bios when the disk is inserted).

Example log:

Arcade ROM directory not found
cdvdman Init

IOP Realtime Kernel Ver.0.9.1

    Copyright 1999 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
loadmodule: fname rom0:SIO2MAN args 0 arg 
loadmodule: id 25, ret 0
loadmodule: fname rom0:MCMAN args 0 arg 
loadmodule: id 26, ret 0
loadmodule: fname rom0:DAEMON args 0 arg 
loadmodule: id 27, ret 0
check card routine start
Get Reboot Request From EE
------------
Update rebooting..

Arcade ROM directory not found
cdvdman Init

IOP Realtime Kernel Ver.0.9.1

    Copyright 1999 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
loadmodule: fname rom0:SIO2MAN args 0 arg 
loadmodule: id 25, ret 0
loadmodule: fname rom0:MCMANO args 0 arg 
loadmodule: id 26, ret 0
read/write allocate memory 4000
open name mc1:SEDCOPY flag 1 data 21170
open fd = -6
Get Reboot Request From EE

Update rebooting..
------------
Arcade ROM directory not found
cdvdman Init

IOP Realtime Kernel Ver.0.9.1

    Copyright 1999 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
loadmodule: fname rom0:SIO2MAN args 0 arg 
loadmodule: id 25, ret 0
loadmodule: fname rom0:MCMAN args 0 arg 
loadmodule: id 26, ret 0
read/write allocate memory 4000
open name mc0:SEDLOAD flag 1 data 21170
open fd = 2
open name mc0:TK4DATA flag 1 data 21170
open fd = -2
buff = 1400000
open name mc0:SEDGAME flag 1 data 21170
open fd = 2
buff = 400000
Get Reboot Request From EE

Update rebooting..
------------
Arcade ROM directory not found
cdvdman Init

IOP Realtime Kernel Ver.0.9.1

    Copyright 1999 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
 Get Reboot Request ... Ignore
loadelf: fname ac0:SEDLOAD secname all
loadelf version 3.30
Input ELF format filename = ac0:SEDLOAD
0 00100000 00374800 ........
Loaded, ac0:SEDLOAD
start address 0x100008
gp address 00000000
Get Reboot Request From EE
Update rebooting..
------------
Arcade ROM directory not found
cdvdman Init
create sema hakama

IOP Realtime Kernel Ver. 2.2

    Copyright 1999-2002 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
check card routine start
loadmodule: fname rom0:SIO2MAN args 0 arg 
loadmodule: id 28, ret 1
loadmodule: fname rom0:MCMAN args 0 arg 
loadmodule: id 29, ret 1
Get Reboot Request From EE

Update rebooting..
------------
Arcade ROM directory not found
cdvdman Init
create sema hakama

IOP Realtime Kernel Ver. 2.2

    Copyright 1999-2002 (C) Sony Computer Entertainment Inc. 

Reboot service module.(99/11/10)
Load File service.(99/11/05)
Multi Threaded Fileio module.(99/11/15) 
iop heap service (99/11/03)
check card routine start
loadmodule: fname rom0:SIO2MAN args 0 arg 
loadmodule: id 28, ret 1
loadmodule: fname rom0:DAEMON args 0 arg 
loadmodule: id 30, ret 0
acfpgald:wait: TIMEDOUT 00 20
acfpgald:wait: TIMEDOUT 00 10

After that, it will get a black screen.

987123879113 commented 1 year ago

The MagicGate memory card stuff is the least of the worries for Python 1 at least. I don't really care about the Namco stuff so I haven't looked at that. For Python 1 to work, Firewire emulation must be improved and then an additional CPU emulator (for the Toshiba TMPR3927 CPU, or maybe someone can try HLEing it if they didn't want to implement the CPU) are required because it handles a lot of things on the Python 1 I/O board, most important being the networking and HDD/CF card.

I have no interest in doing any of the above but if someone wants to send a PR with it I'd take a look at it.

Python 1 stuff fails predictably where you would expect even if you get it booting without the Firewire device.

AKuHAK commented 1 year ago

After committing https://github.com/987123879113/pcsx2/commit/1146175648b9e2d5350fb9b5fb83afe32081d781 MagicGate stopped working on memory cards. Just a note for those who were using your fork for FMCB development, those should stick with artifacts from commit c54d20753a9bcc4bb3921b27617cde260505b2ff

987123879113 commented 1 year ago

I mentioned it on Twitter but I'm willing to accept PRs if someone takes the time to rewrite the SIO magicgate code to work with the new SIO rewrite. It's a feature that is otherwise unused by the intended usage of this fork (Python 2 arcade emulation) so it wasn't an accidental breakage and won't otherwise be restored unless someone else rewrites that code.

AKuHAK commented 1 year ago

Doh, it is so difficult to rebase @Florin9doi fixes. To my mind, it will be much easier to create fixing commits without breaking all forks that are based on his work (anyway I appreciate his work). Anyway, as actions are disabled in his fork, Linux builds are broken now.

987123879113 commented 1 year ago

That's exactly what I had to go through the last time you asked me to sync with Florin9doi's branch. He rebases every time he updates the branch which means all of those commits have new commit hashes and can't be cleanly merged back into an existing fork using his branch. It also ideally requires an audit of all of the code in every commit because it's impossible to tell if anything in a previously merged commit changed or not since it's handled as an entirely new commit (sneakily trying to fix bugs somewhere, etc).

If there's specific commits you want to pull in from his branch then I'd suggest just trying to cherry-pick the specific commits and pray nothing changed in any of the older commits.

AKuHAK commented 1 year ago

@987123879113 did you check if hdd sparse file support (added recently) works well?

987123879113 commented 1 year ago

@AKuHAK I haven't used PCSX2 to make an HDD image in a long time (I've always used my own images) so I haven't tested that feature.

987123879113 commented 1 year ago

Just an FYI but there's a USB rewrite PR upstream and whenever that gets merged it's likely going to significantly break the Python 2 USB device code and require a nearly full rewrite, and I don't really see myself rewriting it a 3rd time any time soon so this fork is probably going to go into a bit of a hibernation. I recommend forking and ripping out the Python 2 code when that time comes if you wish to continue using latest upstream + MG support.

AKuHAK commented 1 year ago

I see, thanks for the information. Anyway, in the upstream Python2 fork MagicGate MemoryCard still is broken: FMCB 1.9 cannot be installed or loaded if installed earlier (while 1.8 installed and loaded just fine). I wasn't able to thoroughly debug the reasons for it so I also hibernated the pcsx2 before the SIO rewrite.

AKuHAK commented 1 year ago

Python 1 stuff fails predictably where you would expect even if you get it booting without the Firewire device.\n\n

Hi. Can you tell what bios image is using python1 (romver, md5)? And also, what did you changed in pcsx2 sources for arcade memory dongles to work?

987123879113 commented 1 year ago

The memory card dongles aren't really important to Python 1 so I didn't bother with it. It does some basic generic init and then loads the game executable from the HDD, unlike the Namco stuff which has the game executable on the memory card.

In my pic I converted the game's executable to an ELF and was launching it directly. Then it dies when it tries to do things with the Firewire registers because it's not implemented and pretty much everything important goes through the Firewire device (HDD/CF access, I/O, dongles, etc).

As for BIOS I was using: https://github.com/mamedev/mame/blob/master/src/mame/konami/kpython.cpp#L244-L245

Unlikely to make much of a difference but here's the small part of WIP code I wrote for the FW emulation that I was hacking away at for Python 1 a while back. https://gist.github.com/987123879113/f4464e645709480b03f1a683badfecf1

AKuHAK commented 1 year ago

@987123879113 artifacts are expired. Is it possible to rerun jobs and upload them into release page?

987123879113 commented 1 year ago

You missed the release added last week. https://github.com/987123879113/pcsx2/releases

987123879113 / pcsx2