Add Pop'n Music support

[icex2]

Self-explanatory.

Some notes:

• Card reader/acio infrastructure is already available and used by other games. Should be recycable.
• IO2 infrastructure also available. This needs a new message layer to handle the inputs/outputs according to how popn hardware is hooked up. The device and vendor layers should be fine.
• Roundplug infrastructure for older popn games might be available already. There are slight differences depending on the games and versions used (e.g. old IIDX games like Happy Sky roundplug vs. jubeat 1). Maybe the current implementations already cover everything we need here but maybe not. Needs to be verified.
• As far as I am aware of, popn didn't change a lot software and hardware wise over the years. So the complexity level of getting all PC based games supported might be fairly low (compared to something like IIDX...).
• Architecture: Might require a multi hook architecture like iidx but likely less complex. In order to determine the requirements, the following approach might be suitable: Support popn 15 and the latest version first. Depending on how much they differ in their hook implementations, we should be able to derive a fitting architecture and also add implementations for all other versions fairly quick.

This issue should be further refined once initial research and prototyping has started.

Some TODOs collection from comments:

• [ ] Fix naming of cyusb3 driver for popn (https://dev.s-ul.eu/djhackers/bemanitools/-/issues/28#note_5020)

[icex2]

Some progress notes: Popn is not as straight forward as I expected because of the following blocker: The ezusb.dll of popn15 is doing IO init stuff in DllMain. However, DllMain cannot be hooked with bemanitools right now which causes IO init to fail (including looking for a real IO). Therefore, bemanitools, or rather the inject tool, needs a feature to hook/block DllMain of ezusb.dll when the executable is loaded into a separate process space.

Potential solutions might utilize hooking LdrpCallInitRoutine (https://reverseengineering.stackexchange.com/questions/6018/how-to-hook-the-entry-point-of-a-dll) or LdrpInitialize (http://www.nynaeve.net/?p=205).

[icex2]

unassigned @icex2

[icex2]

More research notes:

• Replacing the current process doesn't seem do-able, e.g. Linux exec syscall. This would allow us to load the new exe into the current process (easily), mess around with the environment before jumping to the main entry point of the loaded exe. There is a way to use LoadLibraryA to load an executable, but that seems to be quite complicated to achieve: https://stackoverflow.com/questions/19110747/loadlibrary-an-exe
• Injecting another proxy dll to the remote process first, which does some hooking of any kind to prevent ezusb.dll's DllMain from being called or having a proxy ezusb.dll with an empty DllMain to block loading the real ezusb.dll automatically, doesn't work. Once another remote thread is created and run to execute any code from another dll, the main thread leaves the suspended state and loads all required dependencies before the other remote thread is executed (apparently? Might be that both do run at the same time and we even have some timing issues here...).

[icex2]

Finding a solution that just works, no matter the complexity, is really difficult when keeping the requirements of:

• not modifying the original ezusb.dll file
• blocking/hooking of the DllMain call of it

However, if we allow intervening on the file system level, the following solution is very straight forward to implement (I guess):

• The original ezusb.dll in the game directory is renamed to ezusb2.dll or something else
• BT5 provides a proxy stub version of the ezusb.dll that is placed in the game directory
• The stubbed ezusb.dll does not implement anything inside DllMain
• This approach is identical to how BT4 or spicetools is operating
• However, the stub ezusb.dll implements lazy dynamic loading of ezusb2.dll + calling DllMain on any API call and proxies any ezusb API calls to the exports of ezusb2.dll
• This way, the game can load the stub ezusb.dll, the hook module can do the hooking which happens before the game is executed/any calls to the the ezusb API are issued
• Once calls to the ezusb API happen, the real ezusb, ezusb2.dll will be called via the proxy ezusb.dll

[icex2]

We need two different approaches for the different generations of popn games:

• exe based
• dll based

Exe based games need inject.exe to setup a remote process with the executable and the hook dll. This requires the solution described in my previous comment.

Dll based games can utilize the fact that the game is loaded into the same process space as launcher.exe. This might allow us to pre-load ezusb.dll using LoadLibraryEx and mute the call to DllMain. This could be done in a special hook dll that is hooked before the game's dll is loaded into the process space. This avoids the whole re-naming thing with ezusb.dll and ezusb2.dll as described above. Then, setup the rest of the game's process, launch the "normal" hooks. The ezusb proxy takes care of lazy calling the original ezusb DllMain on the call of the first API function of the ezusb API.

[icex2]

The whole renaming issue can be scripted to avoid the manual and error prone work by the user. Make ‘gamestart.bat’ check if a ‘ezusb-proxy.dll’ is available and do the renaming swap:

• ‘ezusb.dll’ -> ‘ezusb-orig.dll’
• ‘ezusb-proxy.dll’ -> ‘ezusb.dll’

[icex2]

Turns out that just using a ezusb-proxy.dll that uses LoadLibraryA to load the ezusb-orig.dll doesn't work. Any hooks to setupapi which is used in ezusb-orig.dll's DllMain method to find a connected ezusb device are not hooked when the setupapi hooks are setup in either pnmhook1 or ezusb-proxy. When ezusb-orig.dll is loaded, it sets up its own import table, obviously, and resolve all the references during LoadLibraryA. You can use LoadLibraryExA with the flag DONT_RESOLVE_DLL_REFERENCES to avoid calling DllMain and resolving the references of the imports. However, that means you have to resolve them manually.

Concluding these tests, one needs to add a pe-loader module to capnhook which provides separate calls to:

1. Load a dll file into memory
2. Resolve imports/dependencies of the loaded dll correctly
3. Call DllMain

Where 3) needs to be a optional when loading and triggerable post loading. This allows one to setup hooking after 1) and 2) for setupapi and then do 3).

A reference implementation for loading DLLs which looks very complete and should help solving the above: https://github.com/gbmaster/loadLibrary

[icex2]

Note regarding different pnmhook versions required: Likely, we need two hook versions, only:

• pnmhook1 for 15-19 (exe-based games)
• pnmhook2 for 20+ (dll-based games)

Furthermore, all versions share a very identical ezusb.dll which means that a proper dll loader solution, as posted above, is required to get any version implemented with BT5.

[icex2]

In GitLab by @shtokopep on Oct 18, 2020, 17:38

(posting in this thread so all pop'n dev support notes are at the same place)

side note about popn15 ezusb.dll: the ezusb.dll bundled with ahn version has unmangled exports, but later datecode versions do have the regular decorated names as the other.

IOBoard preliminary notes:

I've been looking at the original ioboard (my primary goal being spoofing it with an arduino). I ran into this hurdle:

• IO2 board is a Cypress FX2LP board, with the legacy XP driver ( ezusb.sys / ezusbw2k.inf )
• Current driver is CyUsb3.sys, but it's not backwards compatible.
• Legacy driver won't work on newer windows.

Hooking up the original ioboard on a win10 pc, i could install the newer driver, but then pop'n ezusb.dll fails to load, and with an usb packet sniffer I see nothing. According to the programmers reference guide for CyUsb3.sys ( https://www.cypress.com/file/145261/download ), page 56, IOCTLs have changed between ezusb.sys driver and the new one so I guess ezusb.dll fails to find a suitable device, before reaching the usb communication stage.

(apologies if I misunderstood BT5 approach with io emulation, feel free to correct me) It seems to me, if you want popn to work with an unmodded ezusb.dll (and without hooking it as spicetools does), then we need to write a new x86/x64 driver which would then be able to receive messages sent by ezusb.dll and process them however we want

I think this approach wouldn't be suitable as it's constraining for the end user having to install a kernel driver to run pop'n... and on the other end, if you don't go that route, then you need to hook ezusb.dll, then you might as well hook the usbPadRead() and usbLamp() functions, I'm not sure if there's any benefit not doing that...

[icex2]

Hey, very appreciated you reached out to this thread with your concerns. Answering below after quoting various bits from your reply.

side note about popn15 ezusb.dll: the ezusb.dll bundled with ahn version has unmangled exports, but later datecode versions do have the regular decorated names as the other.

Well, that is correct but be careful that the ezusb.dll that is included in the ahnada versions is not the original ezusb.dll. It was either fully replaced or patched to have stuff like keyboard emulation implemented.

Hooking up the original ioboard on a win10 pc, i could install the newer driver, but then pop'n ezusb.dll fails to load, and with an usb packet sniffer I see nothing.

I haven't tested this with Popn's IO2 but it should be identical hardware-wise to the IO2 used with IIDX. The code for the ezusb drivers was included in one of the SDKs and I re-compiled the driver for recent Windows versions. Have a look in bemanitools-supplement, especially here.

(apologies if I misunderstood BT5 approach with io emulation, feel free to correct me) It seems to me, if you want popn to work with an unmodded ezusb.dll (and without hooking it as spicetools does), then we need to write a new x86/x64 driver which would then be able to receive messages sent by ezusb.dll and process them however we want

Fortunately, that should not required (though, I could not test this, yet). See my reply to your other paragraph above. BT5 does system call intercepting which boils down to hooking CreateFileA and DeviceIOControl for ezusb emulation. Have a look at how this is already done for iidx's ezusb2/IO2 emulation. These part can be re-used for popn as well.

• device level hooking and emulation
• iidx message/protocol layer hooking and emulation

I think this approach wouldn't be suitable as it's constraining for the end user having to install a kernel driver to run pop'n... and on the other end, if you don't go that route, then you need to hook ezusb.dll, then you might as well hook the usbPadRead() and usbLamp() functions, I'm not sure if there's any benefit not doing that...

For pure emulation, as explained above, we can be glad that this is not required. The only use-case when you would need the driver is when you want to run the game with real hardware.

I hope that clarifies a bunch of things. Let me know if you have any further questions.

[icex2]

In GitLab by @shtokopep on Oct 18, 2020, 23:36

thank you very much for your reply and the info! :D

Well, that is correct but be careful that the ezusb.dll that is included in the ahnada versions is not the original ezusb.dll. It was either fully replaced or patched to have stuff like keyboard emulation implemented.

yes, maybe the .exe was patched to load unmangled functions and konami never used unmangled versions at all

The code for the ezusb drivers was included in one of the SDKs and I re-compiled the driver for recent Windows versions. Have a look in bemanitools-supplement, especially here.

unfortunately this driver seems to be the same as the CyUsb3 driver which I had tested before :( I tried to install it just in case, but same error, then I checked and indeed both CyUsb3.sys files are identical... Maybe iidx used IO2 on a win7 system? Having another look, it seems like the driver in ezusb (legacy) C01 is the same as the one installed on the popn bemanipc though, i'm gonna try to adapt the x64 version inf and see how that goes..

BT5 does system call intercepting which boils down to hooking CreateFileA and DeviceIOControl for ezusb emulation.

This is very good news, didn't know about that technique! I've had a look at the iidx code you showed me, and indeed if I manage to get my usb sniffer to work on the ioboard I should be able to reverse the protocol and adapt the code to popn case... will keep you updated (if I don't manage to make the driver work, i'll install winxp on another machine to do it (tried through a VM but loadlibrary calls on ezusb.dll fail with "access to invalid address")... so far i'm stuck cause usb sniffers i've tried won't run on the bemanipc, so having the ioboard work on a regular pc install would solve that issue)

[icex2]

unfortunately this driver seems to be the same as the CyUsb3 driver which I had tested before :( I tried to install it just in case, but same error, then I checked and indeed both CyUsb3.sys files are identical... Maybe iidx used IO2 on a win7 system? Having another look, it seems like the driver in ezusb (legacy) C01 is the same as the one installed on the popn bemanipc though, i'm gonna try to adapt the x64 version inf and see how that goes..

That sounds super odd and doesn't align with what I know about these. But, I have used these drivers only with real hardware that is supposed to be used with IIDX. I am also not 100% sure about the (non) differences of an IO2 that is used for IIDX and one that is used for Popn. That is something that I still wanted to look into once I the whole popn hook topic has proceeded.

But, let me know what you can figure out when you are already playing around with it. Any information might help later.

This is very good news, didn't know about that technique! I've had a look at the iidx code you showed me, and indeed if I manage to get my usb sniffer to work on the ioboard I should be able to reverse the protocol and adapt the code to popn case... will keep you updated (if I don't manage to make the driver work, i'll install winxp on another machine to do it (tried through a VM but loadlibrary calls on ezusb.dll fail with "access to invalid address")...

I think you can make your life easier by taking a look at any popn ezusb.dll and how it's called in the game's executable. Since I expect this to be very close to how it is used on IIDX, the code available in BT5 can help you to browse around any disassembly of popn's ezusb.dll. You mainly have to compare what's different and take notes of that. Once that's done, I hope we can re-use the ezusb device layer in BT5 and just have to introduce a new popn message layer on top.

so far i'm stuck cause usb sniffers i've tried won't run on the bemanipc, so having the ioboard work on a regular pc install would solve that issue)

Maybe you might want to look into using and extending the ezusb2 user space driver. There is already plenty of code to get started with this.

• ezusb2 module: Simple module to open a ezusb2 board and establish communicate to load your firmware to it
• ezusb2-tool: This is an actual executable that allows you to open the board an flash it. It is included in the binary distribution under tools.zip: ezusb2-tool.exe. Since we don't have the popn firmware dumped yet, you might want to use this with the iidx IO2 firmware for testing purpose.

[icex2]

In GitLab by @shtokopep on Oct 19, 2020, 14:35

Yes it does seem like despite using the same hardware, they use a different driver, cause CyUsb3.sys is not present on popn system. And since IOCTLs are not the same, and in some case split into several calls in CyUsb3.sys compared to the legacy ezusb.sys driver, looks like there might be a bit more work required on the ezusb emulation layer (although they should be identical to the former ezusb C02/D01 ioctls)

(About the ezusb2 user space driver, I don't have popn firmware either. The way I use ezusb.dll in my previous devs is always the same, I do a LoadLibrary then I call these functions (copying the call sequence from pop'n boot) : usbStart(0) / usbFirmResult() / usbCoinMode(3) )

I have some more testing to do tonight, but so far I'm wondering where the popn firmware is read from. Cause it's either embedded inside ezusb.dll (maybe), or libavs-win32.dll (unlikely as this one is shared with other games iirc), or somewhere else in the system which might explain the bug I've encountered with the C02/D01 driver. I'll have to look into the disassembly of usbFirmResult() function to know more.

In IIDX case, where was the firmware dumped from ?

Legacy driver first test

Using the legacy C02/D01 driver with edited VID/PID works slightly better (loadlibrary works, usbStart(0) works, but then it hangs on usbFirmResult()... I can also launch pop'n via spice without any patch at all (as on a real cab) and it will take me to the self test checks (whereas it crashes before launching the game if i don't have an ioboard or if i use the cyusb3 driver) but then of course it errors out on USB IO (because of usbFirmResult() timeout I guess).

Once again I need to look into usbFirmResult disassembly, and comparing the driver files between what is found in the popn bemanipc and the c01 legacy driver xp version. I should be able to provide more info soon

[icex2]

I have some more testing to do tonight, but so far I'm wondering where the popn firmware is read from. Cause it's either embedded inside ezusb.dll (maybe), or libavs-win32.dll (unlikely as this one is shared with other games iirc), or somewhere else in the system which might explain the bug I've encountered with the C02/D01 driver. I'll have to look into the disassembly of usbFirmResult() function to know more. In IIDX case, where was the firmware dumped from ?

IIDX had the firmware available as a file under the data folder. IIRC it is in some hex format that might be intel hex.

However, it was easier back then to just create a memory map of the firmware image after it was loaded by the emulation layer and make a binary dump to the file. That's how it stayed until today.

Therefore, you first need to get to the point with BT5's emulation that the emulation layer is hooked into one of the popn games and receives the ioctl commands from the game. There is a preprocessor macro that can be enabled that the firmware gets automatically dumped to a file, when the game finishes the firmware download.

Using the legacy C02/D01 driver with edited VID/PID works slightly better (loadlibrary works, usbStart(0) works, but then it hangs on usbFirmResult()...

The C02/D01 driver doesn't make sense however since the ezusb2 board is different and needs a different driver. It might be that some very basic things like enumeration and opening the device works, but things like firmware download and further features fail.

Once again I need to look into usbFirmResult disassembly, and comparing the driver files between what is found in the popn bemanipc and the c01 legacy driver xp version. I should be able to provide more info soon

Again, the driver for the C02 (not C01) boards is not the correct one and support the IO2 board correctly. You definitely have to use the ezusb fx2 driver that is either included in the supplements package or take it from a HDD of a game that supports the IO2 board.

[icex2]

In GitLab by @shtokopep on Oct 19, 2020, 19:47

I was wrong about the driver, while the popn bemanipc does have ezusb.sys/ezusbw2k.inf driver, it is the exact same as the one in C02/D01 in which the relevant vid/pid is commented out.

there's no cyusb3.inf/sys, but there's a CyUSB.sys and cyusb.inf which are indeed used for the ioboard (i confirmed with device manager). It's an older version as well and it doesn't appear to be compatible with the one from bt5 supplements since i couldn't make the ioboard work with that one.

I'm attaching the driver there in case it can help you locate sources from that version in the sdk to compile a win10 version... in the mean time I'm gonna test them on my XP VM, or install XP on a new hdd if that doesn't work...

BemaniPC Pop'n IO2 driver :

cyusb.inf

CyUSB.sys

[icex2]

In GitLab by @shtokopep on Oct 19, 2020, 22:03

Ok, thanks a lot for the info, I made some good progress...

• Got the IOboard working on XP VM with original cyusb.sys driver.

• Got the IOboard working on win10 with cyusb3.sys driver.

using the CyUsb.sys driver from the bemaniPC, I managed to get the ioboard working on my XP VM. I managed to capture the whole communication during the reflash process so I have the firmware dump somewhere in there. Recognized stuff similar to the beginning of iidx firmware, except with "KONAMI POPN" instead of "KONAMI 2DX 14". (oh and thus I confirm the firmware is embedded somewhere either in ezusb.dll or libavs-win32, but not in plain form...) Will try to isolate it, then flash it with ezusb2-tool. (I can then confirm it's working properly using a modded version of my popnforwarder which skips the init sequence and goes straight to the usbPadRead and usbLamp calls) When it's working I'll add the firmware dump to the repo :)

Btw, reason it didn't work with cyusb3.sys driver earlier is not differing ioctls, but just the device name in the driver ("Cypress FX2LP No EEPROM Device" is not correct for ezusb.dll, I changed it back to "Cypress EZ-USB FX2LP - EEPROM missing" in the .inf and it works), which is good news for the emulation layer (and good news for me too as it means eventually my arduino spoof can work on win10/home setups as well)

[icex2]

Ok, thanks a lot for the info, I made some good progress...

Nice, great to hear I could help.

Btw, reason it didn't work with cyusb3.sys driver earlier is not differing ioctls, but just the device name in the driver ("Cypress FX2LP No EEPROM Device" is not correct for ezusb.dll, I changed it back to "Cypress EZ-USB FX2LP - EEPROM missing" in the .inf and it works), which is good news for the emulation layer (and good news for me too as it means eventually my arduino spoof can work on win10/home setups as well)

Nice find. I added that to the OP as TODO to look into when working on popn support and to include a separate inf file for popn in bt supplements.

[icex2]

In GitLab by @shtokopep on Oct 21, 2020, 01:01

I managed to dump the pop'n ioboard firmware. I couldn't test it with my forwarder skipping the init, cause obviously if I do that then ezusb.dll doesn't get the device handle to write to it xD

However I'm 100% sure the dump is correct as I built it from the usb communication log (using setup packets for segment size and offset, and computing the header CRC and total segment count "manually"), and using the same procedure on the usb communication log with iidx firmware dump gave me the correct file as well.

I compared the device descriptors I got after flashing via ezusb.dll or ezusb2-tool and indeed they are the same.

Next step will be to study your iidx code a bit more and hopefully get closer to having ioboard emulation working now I can flash the correct firmware.

In the mean time I started agregating the files on a fork on a popn branch https://dev.s-ul.eu/shtokopep/bemanitools-supplement/-/tree/popn

About the TODO (fix naming of cyusb3), beware that renaming the name in the .inf will cause driver signature to be invalid, so if we can find an official version of the cyusb.sys driver for win7+ that would be better...

[icex2]

Really glad seeing progress on this. I will try to support you where I can with the knowledge I have.

However I'm 100% sure the dump is correct as I built it from the usb communication log (using setup packets for segment size and offset, and computing the header CRC and total segment count "manually"), and using the same procedure on the usb communication log with iidx firmware dump gave me the correct file as well.

How did you actually dump it? For reference, here you can see how BT5 is doing that. However, the requires having a hooking stack, i.e. popnhook, to be setup at least with the most basic stuff.

The ezusb2-tool needs to get a binary (not binary text!) file which has the exact same size as the IO2's RAM (64k IIRC).

Unfortunately, there is a major blocker as posted further up the issue that needs to be resolved. I don't consider it extremely complex, it just needs some proper focus time to get it done right, so it can also be applied to other use-cases.

Next step will be to study your iidx code a bit more and hopefully get closer to having ioboard emulation working now I can flash the correct firmware.

Imo, the right step to move forward. Let me know if you have questions regarding the code.

In the mean time I started agregating the files on a fork on a popn branch https://dev.s-ul.eu/shtokopep/bemanitools-supplement/-/tree/popn

Great. Once you consider this ready, feel free to open a MR.

About the TODO (fix naming of cyusb3), beware that renaming the name in the .inf will cause driver signature to be invalid, so if we can find an official version of the cyusb.sys driver for win7+ that would be better...

Good point. I didn't consider this. Nevertheless, the driver needs some sort of "fixing".

[icex2]

In GitLab by @shtokopep on Oct 21, 2020, 20:12

How did you actually dump it?

I dumped it by dumping the whole USB communication with Device Monitor Studio, then keeping the relevant bits:

• I ran my old pop'n io to keyboard forwarder app, which starts by calling the ezusb.dll usbStart() and usbFirmResult() which flashes the firmware to the board)

• I ran ezusb2-tool to flash the iidx firmware (so i could compare the usb dump with the file contents and reverse the format)

It allows me to see every data exchange between the computer and the usb device (when enumerating and requesting descriptors, when transfering data to/from an endpoint etc..). Each "segment" from the ezusb firmware struct is included in its own single control transfer request. A control transfer in a direction is always comprised of two packets: first the request ("setup packet") and then the data.

The setup packet (in hex) was always something like A0 40 xx xx yy yy zz zz, where xx xx is "value", yy yy is "index" and zz zz is "length".

I dumped everything to a file then put it back in your format, ("value" is what you call "offset"), so each segment becomes : zz zz xx xx [data from the control transfer]

Then I inserted the 4 bytes header (aa aa bb bb) manually : bb bb is the total segment count which I calculated. aa aa is the crc16, I actually copied the code from ezusb2-tool to load the file and display the computed crc.

I copied everything into an hex editor to get the correct binary file.

Lastly I confirmed ezusb2-tool.exe works with my reconstructed firmware dump, and checking the dumped device descriptors after flashing the firmware it seems the board is indeed flashed correctly (I had previous descriptor dumps from when I flashed with ezusb.dll)

(btw iidx ezusb2.bin and popn are much smaller than 64k, they are about 6.4k, but with differing sizes. but my guess is that the value/offset writes block at different places in the 64k RAM)

---

Based on the other stuff you said I guess the next step actually should be to setup the popnhook stack correctly and verify I can dump the firmware this way as I know what I'm supposed to get.

Which one of the iidxhook folders is doing that for iidx? iidxhook7/dllmain.c:107 ? (each iidx5 6 7 8 are each for different versions of the game?)

I'll see what I can do about the major blocker once I understand better how these hooks are setup in iidx case ^^

[icex2]

Phew, that sounds very tedious. The traffic dumping migth still come in handy if we are going to be in need of capturing this from the game actually talking to a real IO.

However, since the most relevant parts of the ezusb2 have already been figured out and tested with iidxhook implementations, I suggest to pause working on that topic for now.

Lastly I confirmed ezusb2-tool.exe works with my reconstructed firmware dump, and checking the dumped device descriptors after flashing the firmware it seems the board is indeed flashed correctly (I had previous descriptor dumps from when I flashed with ezusb.dll)

Ok, cool to hear that you could at least get some positive results from this.

(btw iidx ezusb2.bin and popn are much smaller than 64k, they are about 6.4k, but with differing sizes. but my guess is that the value/offset writes block at different places in the 64k RAM)

Yeah, that might be something we could do as well with the firmware images available on bt-supplement to compress them and get the file size down. However, the file size is small anyway, so it wasn't worth bothering for now. I also don't see any practical use-case to invest time into this as there are other things that are way more valuable.

Based on the other stuff you said I guess the next step actually should be to setup the popnhook stack correctly and verify I can dump the firmware this way as I know what I'm supposed to get.

Yes, that's basically picking up where I left off. I have a branch with a bunch of highly WIP code that already has a very fundamental popnhook framework in place.

However, as mentioned, there is one key feature missing that allows us to properly hook into system calls that are executed from the original ezusb.dll:

Concluding these tests, one needs to add a pe-loader module to capnhook which provides separate calls to:

Load a dll file into memory
Resolve imports/dependencies of the loaded dll correctly
Call DllMain

Where 3) needs to be a optional when loading and triggerable post loading. This allows one to setup hooking after 1) and 2) for setupapi and then do 3).
A reference implementation for loading DLLs which looks very complete and should help solving the above: https://github.com/gbmaster/loadLibrary

Which one of the iidxhook folders is doing that for iidx? iidxhook7/dllmain.c:107 ? (each iidx5 6 7 8 are each for different versions of the game?)

The firmware dumping is not a triggerable feature on normal builds of iidxhook. It needs to be enabled by building BT5 with the preprocessor flag seen here enabled. However, that requires us to have a proper hook setup with ezusb.dll, first.

I'll see what I can do about the major blocker once I understand better how these hooks are setup in iidx case ^^

The general idea applied here is called DLL injection and is a common and well known technique to inject custom code which is stored in a separate DLL file (e.g. iidxhook8.dll) into another running process. Once that's done, it opens up further options to mess around with the loaded process, e.g. detouring system calls which is the basis for nearly everything BT5 is doing: IO emulation, path redirects, directx related stuff etc.

If you are not familiar with these concepts, yet, you might be facing a fairly steep learning curve to get an understanding of how this stuff works and is applied in practice. It's not impossible, but I suggest you have to do some reading on these topics first. Otherwise, it will be difficult to understand what BT5 is doing and how to tackle the task that is required to be done for popn here.

Nevertheless, let me know what you think and if you want to continue on this. I am more than happy for anyone that wants to pick this up and take it as a learning opportunity. I will try to support you where possible.

[icex2]

In GitLab by @shtokopep on Oct 22, 2020, 13:31

usb capture

Oh no, it just sounds more tedious than it really is (I'm used to USB protocol by now), and it is a requirement for me anyways (remember my project is also to make an arduino firmware to simulate popn io2, think acrealio but for popn io board lol, but I'm okay with focusing on bt5 compatibility first as there's no real demand for my other project, and yes I do want to continue on this as I'd love to learn more about all this in a practical way)

firmware images

Sorry if I wasn't clear, i meant this is already the case, if you check your ezusb2.bin it's only 6kb already. ezusb2-tool.exe doesn't take a 64kb image file

hook setup

Sorry I wasn't clear enough, I know about compilation flags and precompiler directives, what I meant is "how is the rudimentary hooking stack setup" done, (as you said, "that requires us to have a proper hook setup with ezusb.dll, first.", the question is where/how is that done with iidx. I think iidx stopped using an external dll for io at some point, so if you could point me to a part of the code where stuff is done on a version which uses an external dll and an ezusb2 board I guess that could help, but in the mean time thanks for pointing to that popn branch, I'm gonna have a look at it and compare with iidx code (does it contain other "pe-loader" modules already so I can see how one implements a new pe-loader module in practice?)

Don't worry I'm familiar with DLL injection and hooking principles, it's just that I didn't know it was possible with syscalls as I thought OS dll were a special case. Also I've never used capnhook so any kind of basic examples for that specific tool would be appreciated if you know some :)

thanks for the reply as always, much appreciated :)

[icex2]

Oh no, it just sounds more tedious than it really is (I'm used to USB protocol by now), and it is a requirement for me anyways (remember my project is also to make an arduino firmware to simulate popn io2, think acrealio but for popn io board lol, but I'm okay with focusing on bt5 compatibility first as there's no real demand for my other project, and yes I do want to continue on this as I'd love to learn more about all this in a practical way)

Ok, that is cool to hear that you like that kind of work, not many enjoy this low level stuff. ;) I think something similar to acrealio for the IO2 board that would work with popn but also iidx might be pretty cool and useful even. Also, please don't get me wrong regarding value/priority. Just seeing this from a BT5 perspective and what is probably interesting for the broader audience is having support for popn before digging into more advanced, and for the people working on them, cooler tasks.

Sorry if I wasn't clear, i meant this is already the case, if you check your ezusb2.bin it's only 6kb already. ezusb2-tool.exe doesn't take a 64kb image file

You are right. Then I don't know why I had such a file size in mind. Sorry about the confusion.

Sorry I wasn't clear enough, I know about compilation flags and precompiler directives, what I meant is "how is the rudimentary hooking stack setup" done, (as you said, "that requires us to have a proper hook setup with ezusb.dll, first.", the question is where/how is that done with iidx. I think iidx stopped using an external dll for io at some point, so if you could point me to a part of the code where stuff is done on a version which uses an external dll and an ezusb2 board I guess that could help, but in the mean time thanks for pointing to that popn branch, I'm gonna have a look at it and compare with iidx code (does it contain other "pe-loader" modules already so I can see how one implements a new pe-loader module in practice?)

Ok, I hope I got you right this time. :) Here are some pointers that might help (also personal suggestions, there might be further angles to approach this, of course):

• Start with iidxhook3's main module. Basically, every game we support has one "hook module" as we call them that contains all the setup code to hook various system/library calls using different modules from the internal hook library.
• Some pointers for the hook library:
  • You might want to checkout iohook first as it does hooking of IO related calls, e.g. CreateFileA, ReadFile, WriteFile and especially important for many IO boards, DeviceIOControl
  • Futher helper libraries adding another layer of abstraction, e.g. for rs232, can be found in the internal hooklib library. I don't think you need these for now, but I think it's good to know this is around especially because the setupapi stuff that is used with the ezusb boards, is handled there in a separate module.
• Regarding iidxhook3/main.c's bootstrapping, the library hooks into the OpenProcess call to setup various hooks, e.g. IO. That function was chosen because it is one of the first system calls by the supported games that we can hook into before any init code by the game is run. Note: We didn't have pre-WinMain hooking options back then. I think capnhook supports this now, but we haven't switched to that so far.
• One of THE important calls to setup IO emulation with the ezusb2 module is this line. It inserts the IRP callback that will be called by the iohook module when the system calls the supported IO related functions, e.g. CreateFileA, DeviceIOControl etc.
• Therefore, the entry point to the first layer, the device layer, of ezusb2 emulation is here
• Everything else is dispatched and handled from there on: The different types of IO system calls, parameters provided to them, calling the real versions of these calls if certain criteria don't match etc.
• Handling of the different IOCTLs starts here. Further handling follows according to the different parameters in the usb data provided by the call. Note: This does not follow the USB standard entirely. You MUST use cypress's own definition of these structures, if you haven't figured that out already.
• On top of the device layer, the message emulation layer for iidx is built. It is set up here in the iidxhook module. This includes various callbacks for reading and writing the ezusb firmware defined endpoints.
• Everything else from this point on is figuring out the data structure of the packages exchanged with the game.
• Depending on what the game specific firmware implements, this can be simple data exchange but also contain some kind of "rpc" call backend for the different sub-systems, e.g. eeprom, security, 16seg display etc. For IIDX, you can find the structure that is set up for that in the message emulation layer here. The separate "node" modules that handle the different endpoints can be found here. These are shared with the ezusb emulation layer. For example, node 16seg is one of the simpler ones that returns 16seg data for IIDX.

I hope these pointers help you to lurk around the code and understand the various layers and modules better. Again, don't hesitate to reach out to me regarding further questions.

[icex2]

In GitLab by @shtokopep on Oct 25, 2020, 09:53

thank you very much for these pointers this is exactly the kind of info I was looking for :) (funny I recognized the whole firmware update process among these :D) will study that some more and compare with your popn branch then i'll try to get to that firmware update state in popn case!

[icex2]

New code reference that might help solving the ezusb.dll library loading problem: https://gitlab.com/batteryshark/vx

According to the author, this uses queued apc and therefore also hooks syscalls like ldrloaddll (not LoadLibraryA). This should provide more control over the process that is loaded. Kernel32 isn't even loaded at that point and doesn't even load with the project above. The latter might be a bit too much for bemanitools, but the available code should have what we need for popn.

[icex2]

Another reference with some pointers: https://github.com/batteryshark/VXBootStrap

this would be what you want then
it's the earliest and out of the thousands of things I've hooked with it
I've never had a problem
just remember, you're hooking right after ntdll maps so anything else you have to bring with you
like kernel32 wont be fully mapped yet
so you have to import that if you want it

[icex2]

In GitLab by @shtokopep on Jan 7, 2022, 24:56

Here's the promised write up regarding the ezusb2 popn firmware as I've had a hard look into it while writing a (mostly)compatible firmware for arduino lol :)

from init to firmware flash (TL;DR ezusb2-emu looks like it will handle it entirely without changes)

it has two different device/configuration descriptors

first one is the default one (before firmware is flashed), it's a weird descriptor with one interface, no endpoints exposed, but still 3 alternate settings which expose 6 endpoints each. Not including them here cause fortunately for us they are not actually needed anyways, ezusb.dll will send firmware flash packets and go through with the procedure even if we are already exposing the phase2 descriptors.

Firmware flashing procedure at usb level is obviously the same as with iidx so I'm not detailing here.

Upon firmware flashing, the device reenumerates by itself with a slightly different device descriptor,

const uint8_t PROGMEM DeviceDescriptorPhase2[] =
{
  0x12,        // bLength
  0x01,        // bDescriptorType (Device)
  0x00, 0x02,  // bcdUSB 2.00
  0x00,        // bDeviceClass (was 0xFF in phase 1)
  0x00,        // bDeviceSubClass (was 0xFF in phase 1)
  0x00,        // bDeviceProtocol (was 0xFF in phase 1)
  0x40,        // bMaxPacketSize0 (64)
  0xB4, 0x04,  // idVendor (0x04B4)
  0x13, 0x86,  // idProduct (0x8613)
  0x00, 0x00,  // bcdDevice (0, was 160.1 in phase 1)
  0x01,        // iManufacturer (1, was 0x00 in phase 1)
  0x02,        // iProduct (2, was 0x00 in phase 1)
  0x00,        // iSerialNumber
  0x01         // bNumConfigurations 1
};

and a new configuration descriptor

const uint8_t PROGMEM ConfigurationDescriptorPhase2[] =
{
  0x09, 0x02, 0x2E, 0x00, 0x01, 0x01, 0x00, 0x80, 0x32, // Config
  0x09, 0x04, 0x00, 0x00, 0x04, 0xFF, 0x00, 0x00, 0x00, // 1 Interface
  0x07, 0x05, 0x81, 0x03, 0x40, 0x00, 0x01,             // 4 endpoints  0x81 IN  Interrupt
  0x07, 0x05, 0x01, 0x03, 0x40, 0x00, 0x01,             //              0x01 OUT Interrupt
  0x07, 0x05, 0x02, 0x02, 0x00, 0x02, 0x00,             //              0x02 OUT Bulk
  0x07, 0x05, 0x86, 0x02, 0x00, 0x02, 0x00              //              0x86 IN  Bulk
};

(note: endpoints configuration is consistent with what is shown in ezusb2-emu)

which gives us the following device path \?\USB#VID_04B4&PID_8613#6&ce0d991&0&3#{a6782bce-4376-4de2-8096-70aa9e8fed19}

ezusb.dll will then pause for a little while (10sec?) and attempt to read the Manufacturer string proceed if it's indeed "KONAMI" (already handled in https://dev.s-ul.net/djhackers/bemanitools/-/blob/master/src/main/ezusb2-emu/device.c#L310 ) (note: the Product string is "POPN" but the game never attempts to read it)

Up until this point, everything seems to be handled properly by ezusb2-emu.

Pop'n firmware

This is where we get to the popn specific part

communication workflow

1. (discovery and firmware flash)
2. host reads manufacturer string
3. initialize
   • poll interrupt endpoint 0x81 (gets the "initial reply")
   • poll interrupt endpoint 0x01 (writes 64 0x00 bytes)
   • poll bulk endpoint 0x86 (gets 512 0x00 bytes)
   • poll bulk endpoint 0x02 (writes 512 0x00 bytes)
   • poll bulk endpoint 0x02 (writes 512 0x00 bytes) (yes this happens twice)
4. I/O polling loop:
   • poll interrupt endpoint 0x81 (gets input report)
   • poll interrupt endpoint 0x01 (writes the "initial command", then writes 64 0x00 bytes, then finally loops with "output report" command, and also sends the "mystery command" once after 80 output reports, then never again)

(note: the IO polling loop timing is not precise as it's handled by OS threads, but it hovers around 3-4 0x81 polls for one 0x01 poll)

Bulk endpoints

Any transfer from/to these endpoints is of 512 bytes which are all 00.

Also they only happen during the initialization phase and are never used again once the I/O polling loop has started

Interrupt endpoints

Device to Host (IN)

Initial reply

Right after sending the "KONAMI" usb string, the board will send a first packet which can look like this :

f4 3b 4e f7 d7 fb 7d fd ff c8 bf 7f 7f 7d df e7
cb fb ff ed b9 7b 07 ff fb a5 75 b7 bf 2d c6 ff
e7 9b fb 77 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf
b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f

iirc the first 36 bytes are always the same, not sure (anyways always sending that exact 64 byte as our first packet will work)

last 28 bytes change a little between unplug/replug, mostly similar, here are 3 examples :

            30 fe d3 5f 9e 6e df 96 ae d7 f3 df
b7 e8 37 ef 11 2b dd f6 dd 3f d8 e7 dc ce 31 7f

            32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf
b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f

            32 7e d3 5f 9c 6a d6 96 af d3 f3 ff
97 c8 37 ef 14 2b dd f7 dd 3f d8 f7 dc ee 33 7f

Doesn't seem important either, I'm not sure if it's actually checked by the game or if it's just an artifact caused by the board not updating the last bytes (todo: check if it still works when I change them during runtime)

struct {
  uint8_t  header[36];    // f4 3b .... fb 77
  uint8_t  footer[28];    // 28 bytes that change on each unplug/replug
};

Input report

once the initial reply has been sent, all the other reads will follow this format :

03 1d 85 ** ff 00 ** fd S1 S2 S3 S4 00 7d df **    
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
** ** ** ** 32 7e d3 5f 9c 6a d6 96 af d3 f3 ff
97 c8 37 ef 14 2b dd f7 dd 3f d8 f7 dc ee 33 7f

where ** denote the variable bits, and S1 S2 S3 S4 the input bitfield (more on that later).

Here's what I reversed so far

typedef struct {
  uint8_t  header[3];         // always 0x03 0x1d 0x85
  uint8_t  unk;               // random counter? or checksum??
  uint8_t  const_ff00[2];     // 0xff 0x00
  uint8_t  coin_count;        // coin count
  uint8_t  const_fd;          // 0xfd
  union {
    uint32_t full;
    struct {
      uint8_t  ts;            // test/service bitfield
      uint16_t button;        // button bitfield
      uint8_t  dip;           // dip switches
    } split;
  } bitfield;
  uint8_t  const_007ddf[3];   // 0x00 0x7d 0xdf
  uint8_t  seqnum;            // internal sequence number
  uint16_t history[10];       // button bitfield history
  uint8_t  footer[28];        // 28 bytes that are always the same but different each boot
} input_report_t;

couple extra words about selected fields

unk

unk changes at every single transfer, and doesn't appear to be a checksum (varies too much with a small increment of only one byte elsewhere, and then you can also observe twice the same packet with just the unk byte being different).

                        00000000   03 1d 85 ea ff 00 00 fd ff ff ff f7 00 7d df 4f    .............}.O
                        00000010   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
                        00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
                        00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

                        00000000   03 1d 85 19 ff 00 00 fd ff ff ff f7 00 7d df 50    .............}.P
                        00000010   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
                        00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
                        00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

                        00000000   03 1d 85 3f ff 00 00 fd ff ff ff f7 00 7d df 51    ...?.........}.Q
                        00000010   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
                        00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
                        00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

                        00000000   03 1d 85 6e ff 00 00 fd ff ff ff f7 00 7d df 53    ...n.........}.S
                        00000010   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
                        00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
                        00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

Fortunately it's not important for us, I'm keeping it constant in my implementation and the game doesn't complain.

coin_count

funny note: increment of this field is what triggers the insert coin sound, rather than the coin mech bit being set in button bitfield...

bitfield

bitfield is a uint32_t following the exact same format as usbPadRead() unsurprisingly, however it's active low unlike usbPadRead (just like inverted_pad in ezusb2-iidx-emu seems to be)

MSB                                 LSB
    S4       S3        S2        S1
1111 1111 1111 1111 1111 1111 1111 1111
     ----  -      ----------- --     
    dip4-1 C         sw9-1    TS

(Note: the bytes actually appear in S1 S2 S3 S4 order in the interrupt transfer but this is just endianness at work...)

C is coin mech, T is test, S is service, sw9-1 are buttons 9 to 1, dip are dip switches (note: technically the reset button is still somewhere in the S1 byte but it's unused in bemanipc hardware)

I'm using a union to be able to easily access the "button" middle part as it is important later (cf. history)

seqnum

seqnum is an internal sequence number. Looks like the ezusb2 is running its own state polling loop, then when an interrupt poll comes from the host it will give the latest status (which means sometimes it happens that this seqnum skips one beat in which case you notice the history also got an extra update, or it can also happen that this number doesn't increase at all between two reads, in which case the rest of the packet also didn't update)

history

history is interesting as it contains in order the 10 previous button states (only the button/coin bitfield, no dip or test/service).

here's an example where you can see the state "fe ff" propagating on the second line along the seqnum increases

00000000   03 1d 85 7f ff 00 00 fd ff fe ff f7 00 7d df 03    ............}..
00000010   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

00000000   03 1d 85 ae ff 00 00 fd ff fe ff f7 00 7d df 05    .............}..
00000010   fe ff fe ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

00000000   03 1d 85 c0 ff 00 00 fd ff fe ff f7 00 7d df 05    .............}..
00000010   fe ff fe ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

00000000   03 1d 85 d4 ff 00 00 fd ff fe ff f7 00 7d df 06    .............}..
00000010   fe ff fe ff fe ff ff ff ff ff ff ff ff ff ff ff    ................
00000020   ff ff ff ff 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf    ....2~.O.j......
00000030   b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f    ..7..+...?....1

I suspect this history is actually being processed by ezusb usbPadReadLast() function, probably for debouncing? it would definitely be worth having a look at disassembly to find out.

Note: you don't actually need to process the history, just updating the bitfield and increasing seqnum is enough to get the game to process your inputs (or at least just updating the bitfield and history[0] is enough (while increasing seqnum by 1 is still ok)).

Host to device (OUT)

Initial command

The very first write to the 0x01 out endpoint is made of only 0x00 bytes during the initialization, then this "initial command" is sent

 00 00 09 03 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

then another write made of only 0x00 bytes is done.

Mystery command

After sending 80 "output report" commands, the following packet is sent

 00 00 0c 3c 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

then output reports continue endlessly... doesn't seem important anyways

Output report

Once the first three writes are done, all other writes (except the mystery command) follow this format:

00 00 09 02 00 00 00 ZZ L1 L2 L3 L4 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

where ZZ is a counter which increments once every 16 or so seconds (can be discarded in our implem anyways), and L1 L2 L3 L4 is the lamp bitfield.

typedef struct {
  uint8_t  command[4];   // always 0x00 0x00 0x09 0x02
  uint8_t  padding[3];   // always 0x00
  uint8_t  counter;      // increases every 16 seconds or so..
  uint32_t lamp;         // lamp state bitfield
  uint8_t  footer[52];   // always 0x00
} input_report_t;

lamp

lamp is a uint32_t following the exact same format as usbLamp() unsurprisingly, and is active high just like usbLamp()

MSB                                 LSB
    L4       L3        L2        L1
0000 0000 0000 0000 0000 0000 0000 0000
--------- -    ---- ---- ----    ------     
  sw9-2  sw1    bl   co  side    top5-1

(Note: the bytes actually appear in L1 L2 L3 L4 order in the interrupt transfer but this is just endianness at work...)

sw9-1 are buttons lamps 9 to 1 (technically the game will set the whole L3 high nibble to 0xf rather than 0x8 but who cares) bl is coin blocker (active low: 0xf is OFF) co is coin counter (todo gotta recheck but probably active high too ^^; ) side are side lamps (from MSB to LSB: left1 left2 right1 right2) top5-1 are hi-lamp 5 to 1

dip are dip switches (note: technically the reset button is still somewhere in the AA byte but it's unused in bemanipc hardware)

Minimal emulation workflow

To summarize, here is the minimum that can be done to make ezusb.dll happy

Bulk endpoints

OUT

always send 512 null bytes

IN

read and discard

Interrupt endpoints

OUT

always read bytes 9 to 12 as they are our lamp bitfield (and they 0 when the command isn't 09 02)

IN

first send

f4 3b 4e f7 d7 fb 7d fd ff c8 bf 7f 7f 7d df e7
cb fb ff ed b9 7b 07 ff fb a5 75 b7 bf 2d c6 ff
e7 9b fb 77 32 7e d3 4f 9c 6a d7 96 ae d3 f3 bf
b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f

then keep sending

03 1d 85 aa ff 00 ZZ fd S1 S2 S3 S4 00 7d df SS    
S2 S3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 32 7e d3 5f 9c 6a d6 96 af d3 f3 bf
b3 e9 37 ef 90 2b dd f7 dd 3f de f7 dc ce 31 7f

while following these rules:

• write the new input state in S1 S2 S3 S4
• increment SS each time
• increment ZZ when you read a coin mech input on S3

(not sure if S2 S3 as the first history input was required or not but I remember it working)

Conclusion

That's all for now, I need to actually port my code to another microcontroller cause atmega32u4 only allows unidirectional endpoints (which means I cannot use 0x81 and 0x01 at once... interestingly enough the game won't complain about not being able to write lamp state). If I find more info along the way I'll update this post.

Looks like we just have to write ezusb2-popn-emu following ezusb2-iidx-emu as a template which should be straightforward (I've had a better look at it while writing this and looks like nothing is missing for popn compatibility). I'll take care of it this weekend if everything goes well.

[icex2]

In GitLab by @33c17f40 on May 15, 2022, 11:57

I needed a way to test some early pop'n stuff so I implemented something but it's a WIP and mostly meant as a test. I independently came to the same conclusions as previously stated in the thread (the exact steps and failures seems to have been the same too, funnily).

The solution I ended up coming up with is a little less than ideal when compared to the preferred method of hooking for bt5, because it puts in a global trampoline hook into kernel32.dll temporarily so anything in the program that calls that specific function will see the hook. But here's what my shim DLL looks like. ezusb.dll gets renamed to ezusb-real.dll, ezusb2-popn-shim.dll gets renamed to ezusb.dll in launcher bat. forward.h/forward.c is the file that reads the reads the ezusb-real.dll's functions and acts as the proxy, you already have one of those in a branch from what I saw.

The hooking is done using MinHook since it's what I am familiar with and is pure C so works with the existing build system. It writes a trampoline to a function that gets called early (GetVersionExA here) in this case before loading ezusb.dll.

Overall the lifetime of the trampoline in kernel32.dll is very short. Load ezusb.dll -> install GetVersionExA hook -> immediately load ezusb-real.dll which triggers its DllEntrypoint and starts the process of checking for the ezusb device -> calls hooked GetVersionExA within DllEntrypoint -> if current module is ezusb-real.dll then hooked GetVersionExA uninstalls itself and installs the normal setupapi hooks.

#include <windows.h>
#include <stdint.h>
#include <stdbool.h>

#include <setupapi.h>

#include "ezusb2-emu/desc.h"
#include "ezusb2-emu/device.h"

#include "hooklib/setupapi.h"

#include "ezusb2-popn-shim/MinHook.h"
#include "ezusb2-popn-shim/forward.h"

#include "util/log.h"

#define EZUSB_REAL_DLL_FILENAME "ezusb-real.dll"

static BOOL (*real_GetVersionExA)(LPOSVERSIONINFOA lpVersionInformation);

static BOOL my_GetVersionExA(LPOSVERSIONINFOA lpVersionInformation)
{
    if (GetModuleHandleA(EZUSB_REAL_DLL_FILENAME)) {
        MH_DisableHook(&GetVersionExA);
        hook_setupapi_init(
            &ezusb2_emu_desc_device.setupapi);
    }

    return real_GetVersionExA(lpVersionInformation);
}

BOOL WINAPI DllMain(HMODULE self, DWORD reason, void *ctx)
{
    if (reason == DLL_PROCESS_ATTACH) {
        if (MH_Initialize() != MH_OK) {
            return TRUE;
        }

        if (MH_CreateHook(&GetVersionExA, &my_GetVersionExA,
            (LPVOID*)(&real_GetVersionExA)) != MH_OK) {
            return TRUE;
        }

        if (MH_EnableHook(&GetVersionExA) != MH_OK) {
            return TRUE;
        }

        ezusb2_shim_initialize(EZUSB_REAL_DLL_FILENAME);
    } else if (reason == DLL_PROCESS_DETACH) {
        MH_Uninitialize();
    }

    return TRUE;
}

Still some bugs to work out with the I/O (thanks to Shtoko Pep for the above details and answering questions on Discord) but I have the popnhook1, graphic hooks, security plugs, audio hook, I/O implemented (working and tested for at least pop'n 16) so once I come up with a good method for ezusb.dll I can send over a PR.

Another method I was considering was something like described in this blog post (https://snoozy.hatenablog.com/entry/2020/01/20/033114 Japanese post, sorry) which describes a method of loading a DLL from memory and includes stuff like resolving IAT. I think something like that may allow for something more in line with what capnhook can do. If I can get it working then I'll make another comment.

[icex2]

In GitLab by @33c17f40 on May 15, 2022, 17:22

Update: I got it working without introducing new dependencies.

I added a new function to the hook library, pe_hijack_entrypoint, which takes the filename of the DLL to load will load the DLL with DONT_RESOLVE_DLL_REFERENCES, resolve dependencies (was able to offload a lot of code to the original capnhook code, pe_iid_get_first etc), and then return the entrypoint of the DLL without executing it.

A quick call to set up the shim DLL's proxy functions and hook_setupapi_init before passing through to the real DLL main and it boots up.

I'll send over a PR for pop'n 15 - 18 (maybe 19 if that used an EXE too) once I fix some things up.

[icex2]

In GitLab by @33c17f40 on May 19, 2022, 20:42

Note: uint8_t const_ff00[2]; // 0xff 0x00 The 0xff here appears to be the last msg_status (value returned from process_cmd) of the last ezusb2_popn_emu_msg_interrupt_write. pop'n music 16 doesn't care but pop'n music 17 won't boot past the security check because it sees an 0xff and thinks an error occurred.

The positioning of the byte aligns with what the IIDX ezusb message has as the status byte as well.

[icex2]

!118 didn't add support for all pop'n music games, but the issue is not very focused anymore. Continuing any futher relevant discussions about remaining versions to be supported in https://dev.s-ul.net/djhackers/bemanitools/-/issues/96