ps2dev / ps2sdk

Homebrew PS2 SDK
Other
907 stars 130 forks source link

Compatibility with GameStar compatible network adaptors - do we want it? #49

Closed sp193 closed 6 years ago

sp193 commented 7 years ago

Some clone/compatible adaptors are no longer compatible with our late designs of ATAD and DEV9. Due to the likelihood of there actually being a multitude of models out there by various manufacturers, the model in question here is a 2015-edition of the GameStar adaptor.

We're in 2017 and the PlayStation 2 is no longer a mainstream console. Since the genuine network adaptor was only released in the US and Japan (and in Europe as only part of the PS2Linux kit), the majority of the world did not see releases. There is possibly a group of users around who may end up with some of the poorly-made adaptors, but may not have the means/do not want to replace them. It is evident that at least GameStar used the (then) sources of ATAD and DEV9 to write the logic for their adaptors, changing how our modules worked has caused such adaptors to become incompatible.

Some of the changes in recent years to ATAD and DEV9 (since 2015) include: -Additional check on the ICRC error state. -Moving the enabling/disabling of DMA (on the ATA interface) into the DEV9 callbacks. -Moving the setting up of DMA direction from before the call to ata_io_start to after the call to ata_io_start. -Correction of the ATA device probing code - our homebrew version worked slightly differently from the original, for one reason or another. For example, devices are assumed to be packet devices unless the ATA LCYL and HCYL are both 0. However, our driver used to assume that the connected device is not a packet device by default. -There was a new check on whether the LCYL and HCYL registers would retain values written. I don't know why this was done, but it may be there to prevent false positives. -There is now an additional operation to disable DMA on the SPEED interface, after the call to ata_io_finish. -As with the newer DEV9 modules, DEV9 now has some code to initialize the SMAP to a known state. -LBA is now set in ata_io_start, causing all operations to always have the LBA bit set to 1.

It appears that the 2015 edition of GameStar expects at least the following: -DMA (ATA) must be enabled with the setting of direction (0x86 must be written). Therefore, the DEV9 DMA callbacks cannot be used. -DMA direction must be set up (ata_set_dir) before the call to ata_io_start. -The "SMAP" cannot be initialized correctly with the initialization code within DEV9.

As for the write to HCYL and LCYL, I haven't really figured out why SCEI added it. Perhaps it's there to detect if the device was incorrectly connected (i.e. floating bus).

As for the DEV9 DMA callbacks, I do not really know why they had to implement them because the SMAP and ATA do not share the same registers for DMA. I guess this means that they want to ensure that DMA will be strictly enabled only before the DEV9 DMA channel is activated, and disabled after the DEV9 DMA channel is deactivated. Right now, the compile-time options that I added to ATAD (see the bottom of this post) will only switch between the two designs for enabling/disabling DMA (pre and post 2015). For the earlier design, there is no code for disabling DMA on the ATA device because the original (earlier design) did not have such a thing... but I cannot tell if I should add it or not.

DEV9 now has some code for initializing the SMAP interface to a known state. Perhaps this is a sign that the SMAP interface can be left in an undesirable state.

That being said, is it worth reaching a compromise? We will be letting go of the changes mentioned above. Personally, I wish to make an accurate copy of the SONY module, but that will prevent owners of clone adaptors from using new software. However, it is worth noting that not all clone adaptors may be made compatible again with such changes. Personally, I own a compatible adaptor, which was already incompatible with our ATAD module in 2013 (before any of my changes were made).

If not, then do we want to make another optional variation of ATAD, for projects like OPL to optionally use? That will allow the effects of the compromise to be confined to only such software, but it also means that owners of such adaptors can only use selected pieces of software, like HDPro users.

The changes can be previewed here, from commit 7faf72a4.

sp193 commented 7 years ago

The more I think about it, I think I can see a reason for the DEV9 DMA callbacks. If DMA is enabled for both SMAP and ATA simultaneously, the SPEED may be unable to correctly determine where the incoming data is supposed to go. That could explain why my custom HDLDump servers in 2011 had reliability issues with the PlayStation 2 hanging, with the HDD activity indicator left on.

A long time ago, our homebrew SMAP (PS2ETH) and ATAD modules did not have support for the callbacks, and so DMA was enabled/disabled in the device driver modules before the DEV9 DMA function is used. Incoming frames were handled from within the interrupt context of the SMAP driver too, so it would have been possible to interrupt an ongoing transfer from ATA. This situation would fit the triggering conditions for such a problem if it exists...

Right now, I think I'm starting to lean towards the alternative ATAD module (i.e. atad-legacy.irx) for software to optionally use.

rickgaiser commented 7 years ago

GameStar I live in europe, and somehow we have plenty of original network adapters here... I'm not sure but I think they where sold here, so I don't know why wikipedia mentions something else. I searched but cannot find any sources of them being sold here (or not). The point is, they are available, and cheap. If they are not available in another country, then they are available on for instance ebay. IMO the only adapter supported should be the original. Clone adapters should be compatible with the original adapter, or there will be a risk of not working. It's not the software that needs to be compatible with an unknown number of unknown clones. This is also impossible for developers to test.

That being said, I realize there is a group of users (of an unknown size) that uses this adapter. There are even still clone adapters being sold on for instance ebay. If they are compatible, then there is no problem, if they are not then i think the name of the irx file should represent the difference, like atad-legacy.irx / atad-clone.irx / atad-gamestar.irx. Looking at the code the support is already compile-time, not run-time. So this is already a different irx file, just with the same name.

We should be happy with developers still interested in developing for the ps2 network adapter. We can't expect them to test and support clone adapters. I think in this case it would be best to split support, becouse it seems to be difficult to support them both at the same time, and the runtime is already incompatible. That way developers with an original adapter can develop and test on their hardware, and developers with clones can support their clone.

DEV9 DMA Some time ago I made a new dev9_dma driver. It's made for (and tested in) ps2linux, but I always hoped it's design could be used for ps2sdk also. It's completely non-blocking and works for both ATAD and SMAP. The existing dev9 driver can be modified to use dev9_dma, so existing applications can still use the blocking dev9 driver call for compatibility. Would there be interest in porting the atad and smap driver to make use of this driver? Advantages are less IOP CPU load, and more speed.

Example SMAP driver using dev9_dma driver: https://github.com/rickgaiser/linux-firmware-ps2/blob/master/smaprpc/src/smap_dma.c Example ATAD driver using dev9_dma driver: https://github.com/rickgaiser/linux-firmware-ps2/blob/master/pata_ps2/src/pata_ps2_ata.c

sp193 commented 7 years ago

Thanks for your input. IMO, we live in developed countries, so shipping is somewhat affordable to us in general. It's just recently that I was thinking... and I felt sorry for those who somehow got stuck with a bad adaptor that was branded "SONY". :( I guess, I didn't want to be responsible for wasting their money. But on the other hand, I really dislike the idea that we're being confined by the badly-designed hardware.

About your DEV9 driver, the idea is good for PS2Linux (since everything runs from interrupts), but the (re-)introduction of the semaphore can result in a performance dip for other software that run on the IOP. Originally, our homebrew DEV9 driver waited for DMA completion with semaphore (unlike the official module), but I remember that a performance test with/without the semaphore had a difference of 0.6MB/s. In OPL, the semaphore results in some visible overhead (stuttering in audio) in some games.

But if it's included, I hope that we'll still be able to understand how the original driver was designed to work.

rickgaiser commented 7 years ago

Less semaphores or other task switching calls (like WaitEventFlag) is good for performance. But I don't think we need a semaphore at such a low level at all. Looking at the code I see this (correct me if I'm wrong): level 1: ata_io_finish calls ata_dma_complete and WaitEventFlag level 2: ata_dma_complete calls dev9DmaTransfer and WaitEventFlag in a loop untill all blocks are received level 3: dev9DmaTransfer starts the transfer, and then enters a busy waiting loop

I think ata_dma_complete can be modified to start dma and interrupts and then wait for completion only once. The DMA completion interrupts can then be used to start the next DMA transfer (instead of the loop in ata_dma_complete). Untill the entire request is completed, then ata_dma_complete can be woken up once. This should couse less task switching, and also no more busy waiting, freeing the IOP to do other things. Performace can be further improved by reading larger blocks (readahead), like for instance reading 4K blocks instead of 512byte would couse 8 times less sleeping and waiting. But I don't know what block sizes are used now.

I would like to test thise theories, but I have no way of comparing the changes. Is there something I can use to test performance?

sp193 commented 7 years ago

If the wait condition has been satisfied, WaitEventFlag will not cause a context switch.

What about other drivers like SMAP? For such a driver, we have to transfer data to a buffer first before marking its descriptor as valid and ready for transmission.

If you replace or remove the semaphore, then the way that threads are scheduled can change. We have been using the wrong type of semaphore in a long time (should have been set to THPRI), but that was corrected.

Since the needs for your modules are fairly different from the normal PS2SDK modules, why not just commit it as its own module? I know that in the traditional PS2Linux kit, DMA was done by DMARelay.irx, which only relayed DMA transfers to/from the EE. Do you not perform initialization of DEV9 on the EE anymore? If initialization is no longer done on the EE, does this mean that there is no longer PCMCIA support?

The block size of PFS is 1KB, as in Linux. Zone size (Cluster) is typically 8KB.

rickgaiser commented 7 years ago

I see, so there is zero task switching and 100% busy waiting? So when the maximum speed is about 10MB/s (random io on spinning UDMA66 disk) and the actual speed is 5MB/s, this would mean the IOP spending 50% of it's total time inside a busy waiting loop? I know this looks like the fastest way to do data transfers, but the IOP has a lot of other things to do as well. Like controllers, audio, network, communicating to the EE, etc... This will all lock up during data transfer, just to get 0.6MB/s of extra speed caused by task switching latency.

The same goes for SMAP, but those drivers are way more complex.

What was wrong with the semaphores?

At a low device specific level the needs for ps2linux and ps2sdk are exactly the same:

In linux the dev9 code is entirely in the IOP. dmarelay.irx was never really used by linux 2.6+. I only used it during testing to find out it wasn't going to work fast. From the EE side, the dmarelay was like a blocking read or write. What I needed was an async read and write, with buffering on the IOP side. Unfortunately I do not have a PCMCIA ps2 to test on, so I do not know. They where never sold in europe. Just as I do not have a PSX to test on... they are both on my wishlist though ;-).

2017-07-26 7:30 GMT+02:00 Liu Woon Yung notifications@github.com:

If the wait condition has been satisfied, WaitEventFlag will not cause a context switch.

What about other drivers like SMAP? For such a driver, we have to transfer data to a buffer first before marking its descriptor as valid and ready for transmission.

If you replace or remove the semaphore, then the way that threads are scheduled can change. We have been using the wrong type of semaphore in a long time (should have been set to THPRI), but that was corrected.

Since the needs for your modules are fairly different from the normal PS2SDK modules, why not just commit it as its own module? I know that in the traditional PS2Linux kit, DMA was done by DMARelay.irx, which only relayed DMA transfers to/from the EE. Do you not perform initialization of DEV9 on the EE anymore? If initialization is no longer done on the EE, does this mean that there is no longer PCMCIA support?

The block size of PFS is 1KB, as in Linux. Zone size (Cluster) is typically 8KB.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ps2dev/ps2sdk/issues/49#issuecomment-317951939, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfm30ockxYvhGnVDl0Fjs-61nnPLVqIks5sRs7hgaJpZM4Ogtp4 .

sp193 commented 7 years ago

I am pretty sure that there are no savings from using a semaphore, due to the frequency of frames entering and exiting the system via DEV9 DMA. My concern is more for SMAP, not ATA, since SMAP will make considerably shorter transfers that are way greater in numbers. Yes, I did no tests on this because that was around 2012, when I had not cloned the thread monitor. But sometime ago, I did a series of experiments on why our LWIP stacks (v1.4.1 then) were considerably slower than than the custom one from SMS. The TCP thread had nearly 100% CPU utilization and throughout was much lower (1.2MB/s vs something like 1.8MB/s). The cause turned out to be the use of the kernel message-box functions. Unlike SMSTCPIP, which used its own custom message-passing functions. Getting SMSTCPIP to use SendMbx/ReceiveMbx resulted in equally sluggish performances.

The Mbx functions have a similar design to Semaphore and Event Flag functions. They disable interrupts and perform a number of sanity checks before doing what they were designed to do. Therefore, it is probably the lack of processing power by the IOP that is the real bottleneck. In LWIP v2.0.0, this is no longer really a problem because they got rid of the need to pass messages.

Yes, not reliquishing control of the IOP will mean that the TCP thread is a resource hog, but it does not totally prevent execution of the other device threads if we schedule things correctly with priority. Actions that take place from within interrupts (i.e. SIFCMD, communicating with CD/DVD) will be also exempt from CPU hogging.

I see, so PS2Linux is now deviating from its original design (by SCEI). I agree, that is good.

sp193 commented 7 years ago

Why don't we merge the code for DEV9, but as a compile-time option? We can generate two versions of dev9.irx: ps2dev9.irx (normal version) dev9-advance.irx (advance version with your custom functionality)

You will then always get the latest updates for dev9, but yet the standard functions will remain compliant with their specifications.

rickgaiser commented 7 years ago

DEV9 DMA

You will then always get the latest updates for dev9

I'm already getting the latest updates becouse I'm loading both. The only problem with this is that dev9DmaTransfer cannot be used.

I think I can live with a compile-time option :-). But preferably have both options with the same interface:

default option:

advance/async option:

GameStar How do you feel about splitting the device specific part from atad.irx into a separate modules? One for the original network adapter, and one for gamestar, and possibly others for other clones?

sp193 commented 7 years ago

Yeah, they can have the same interface. Didn't you already have something that already works? I know it was sometime ago when you showed it to me, but perhaps it can be advanced and merged. Would you be doing this, or would you prefer I do it?

Regarding the GameStar: I'm leaning towards making a special version for compatible adaptors of 2015, which can be optionally used instead of ATAD (although it will open up any possible risk of using the older design). It's not meant to be a GameStar-specific thing although the test subject was a 2015-edition of GameStar, so I would like to name the build atad-legacy... although that can be discussed. I'm still in a discussion with wisi about this, in case he has a better suggestion.

Due to how there can be an infinite number of clones, I think I would like to just support the more popular ones that do work quite well. Personally, I do own one of these compatible adaptors, but it was already incompatible in 2013. It's incompatible with HDLoader v0.8c, but compatible with "Free HDLoader". Disabling 48-bit LBA was a workaround for it, but the adaptor was still so bad that it couldn't be used to write sectors.... :(

rickgaiser commented 7 years ago

I'll make the changes, but dont't expect anything soon.

I just bought a GameStar network adapter on ebay. I was surprised to see it has a SATA interface, nice! But a little disappointed to see this "network adapter" does not have a network port (only a yellow sticker covering up the fake port).

Really curious to see if this thing can do UDMA100 ;-).

sp193 commented 7 years ago

Yeah, there is no rush for this project. Thanks for your input earlier as well.

Also, all the best to you with your experiments! And also to getting a good edition of the GameStar.

Yeah, the compatibles/clones have SATA versions. My own compatible is a SATA network adaptor, which contains a FPGA (for the SPEED logic) and a IDE to SATA bridge (JM20330) device.

No known clone/compatible has a functional SMAP interface. I am personally not surprised, since they would need to somehow get a valid MAC address for it. Or cut corners (i.e. issue arbitrary MAC addresses), which could result in potential MAC address conflicts if the console connects to the Internet.

Jay-Jay-OPL commented 6 years ago

@SP193 more reports on the Network Adapter issue.

It seems that the new changes made also affect an official $ony Network Adaptor that may have a SATA modification.

This reply here kind of sums it up: http://www.ps2-home.com/forum/viewtopic.php?p=19720#p19720 -- the member there lists almost all the possible setups with Network Adaptors that have an issue. Keep in mind, they all worked fine with older OPL versions prior to the changes you made.

So far, some reports are: Cloned/Fake Adaptors fail to list or run PS2 games using an internal HDD. But then there are some with an official $ony Network Adaptor can list/play PS2 games, but can't launch POPSTARTER ELFs/VCDs via the APPS page (conf_apps.cfg). Or if they have an official Network Adaptor but has a SATA modification.

So far we have three threads at ps2-home that talk about almost the same subject: (If you have time, please read them thoroughly--I only added a quick summary below--but I'm sure I may have missed a point or two.

Thread 1: http://www.ps2-home.com/forum/viewtopic.php?f=30&t=3450 <-- this link was the primary one that we started to provide him Beta Tests -- but another member jumped in around Page 6 stating that he has a Gamestar Network Adaptor but he doesn't have any problem. And after making him test something, he then was able to report an issue: http://www.ps2-home.com/forum/viewtopic.php?p=19683#p19683

Thread 2: http://www.ps2-home.com/forum/viewtopic.php?f=19&t=3594 <-- this one is the issue reported about not being able to load POPSTARTER ELFs/VCDs from internal HDD using an official Sony Network Adaptor using a SATA modification. But it works fine with older OPL versions prior to the changes you made.

Later in that thread, I compiled the latest ifcaro repo [Open PS2 Loader 0.9.3-1008-Beta-751e52f] for him to test using the APPS page and also using _confapps.cfg file to list his ELFs. And it still failed.

I know you are not in favor of loading POPSTARTER ELFs on OPL, but in this case, it shouldn't fail using even the ifcaro APPS page. So I figure it must be the new changes made to the internal HDD / Network Adaptor and etc.

Thread 3: http://www.ps2-home.com/forum/viewtopic.php?f=13&t=3603 <-- user reports an issue with one particular game -- also using an official Sony Network Adaptor with a SATA modification -- though we can still perhaps rule out that his rip isn't 100% good (i.e. MD5 checksum).

sp193 commented 6 years ago

The problem was with POPStarter itself: http://www.psx-place.com/threads/open-ps2-loader-v0-9-3.13415/page-4

It likely appeared when belek666 "corrected" the way OPL passes the path to HDD-based targets.

Any issues related to changes in game behaviour may be due to some other change that is not related to this thread. I will check through the commits for issues with the EE core, since it may be related to the recent EE patch additions.

Jay-Jay-OPL commented 6 years ago

@sp193 , you know, I also thought that it might have been belek666 code also. I think I did mention it before, that I wasn't sure if it was your code or his code, because both code changes were done almost the same days.

Whoa! I wasn't expecting that! Thanks for the link from psx-place.

I gotta tell you man, how far will this OPL/POPSTARTER drama go? I know that you and I already buried the hatchet with this. At the very least we've been cordially with each other after that issue. You and I still talk about OPL code and stuff.

I think you and I should make a formal announcement that you and I simply buried the hatchet. I think you and I need to do something about this before it even gets worse. It's really petty stuff. And we all know that we will never add any illegal code to OPL in order to launch the VCDs straight from it. Simply loading the ELFs shouldn't really be a problem, because that is what also the APPS page does.

Let me know if you have any ideas how we should handle this. I am already at that point that enough is enough with this petty drama. Just look at how they are now twisting this that my extra page is causing a stoppage to @rickgaiser pull requests. The last thing I want to do is discourage any new developer. I was simply reporting a symptom to him in that ticket: https://github.com/ifcaro/Open-PS2-Loader/pull/59#issuecomment-321240888 -- and he was gracious enough to help.

barronwaffles commented 6 years ago

There is a place for this discussion and the comment section of a github pull request isn't it.

BatRastard commented 6 years ago

I have an official Sony NA that I modified with a SATA conversion kit. No problems on my end - booted Suikoden V a few days ago for a friend. I don't do the Popstarter stuff though - I got ePSXe for that. --Bat--

Get Outlook for Androidhttps://aka.ms/ghei36


From: barronwaffles notifications@github.com Sent: Wednesday, August 9, 2017 9:40:43 AM To: ps2dev/ps2sdk Cc: Subscribed Subject: Re: [ps2dev/ps2sdk] Compatibility with GameStar compatible network adaptors - do we want it? (#49)

There is a place for this discussion and the comment section of a github pull request isn't it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ps2dev/ps2sdk/issues/49#issuecomment-321258311, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADvT0PO32eypGHcSDroHQa4Qz3wtwIQKks5sWbbbgaJpZM4Ogtp4.

sp193 commented 6 years ago

To end this, I think I will follow through with wisi's suggestion... which is to just go for compatibility. The actual workaround only involves a few checks anyway. I cannot test my work, but the patches are in commits ea60bff and 6d2cb98.

Thank you all, who have participated in this discussion.

I'll close this issue. If anyone has a better way, suggestions, comments, please feel free to comment here and I will re-open this issue.

sp193 commented 6 years ago

I selected the wrong registers, so I replaced the two commits mentioned earlier. They should be offset 0x00 (SPD_R_REV) and 0x04 (SPD_R_REV_3). New commits are 48a5a349 and aa1e223.