[FEATURE] smb2 share support

[sahlberg]

SMB2/3 network filesystem support would be very nice to avoid having to use clunky usb sticks to copy files to/from the ps2.

Libsmb2 is a high performance, small footprint smb2/3 client that can be ported to almost any platform that has berkley socket support. It supports up to smb3.1.1 as well as signatures and encryption. It interoperates with Windows, MacOS, Samba, EMC, NetApp, linux-cifsd kernel server and Azure. https://github.com/sahlberg/libsmb2

I have ported libsmb2 so it builds on PS2 EE and it works. I have build small test applications based on ps2sdk/samples/network/tcpip-basic to connect to shares and access them.

It would be awesome to have smb2 support in the launchelf filemanager. I tried to do it myself, copying code from working examples into wlaunchElf with no success, I can't even get the ethernet link to become active, and not even getting to calls to socket() to succeed when I try to put this code into wlaunchElf :-(. So this is way too hard for me to do, which makes me sad :-(

But if anyone wants to attempt it, I can share the setup code I have so far, working examples based on ps2sdk/samples/network/tcpip-basic and any help you would need for the libsmb2 api. I willing to work with you.

I think it would be really really nice to have smb2/3 support to be able to copy files from/to modern windows and other smb2/3 server.

[AKuHAK]

Yeah it will be cool to get normal smb support. As for now there is only ftp and host support, both very slow and buggy.

[sahlberg]

I am willing to try to add it if someone can help me get the networking configured and working. I have tried but the split between EE and IOP and how wlaunchElf sets up the networking is just beyond my skills. If I can get the network to be configured and socket() work, the rest of the work should not be too hard for me to do. A lot of menial and tedious work, for sure, but not too hard.

[uyjulian]

The filesystem and network drivers are on the IOP side, so you could move libsmb2 to the IOP…

[sahlberg]

The filesystem and network drivers are on the IOP side, so you could move libsmb2 to the IOP…

I have it running on EE successfully in some test applications. But if I need to get it running on IOP in order to use it from wLaunchELF then ok.

But I would need some help. In the ps2 branch in my repo (https://github.com/sahlberg/libsmb2/commits/ps2) I have a version of libsmb2 that compiles with the iop toolchain into an irx. It is not complete however since I don;t know how/what I need to do to make it callable from applications running on the EE. I imagine I will need some glue code both for the iop version of libsmb2 as well as some wrapper on the EE side.

If you or someone could have a look at libsmb2, the ps2 branch and help me get these wrappers done so that I can call the two functions : smb2_init_context() smb2_destroy_context() from an application running on EE then I think I should be able to do the rest.

[sahlberg]

I haven't been able to get anything running with the lwip stack on iop (not even finding any examples that work :-( ) so I have to keep running libsmb2 on EE and talking to LWIP running on EE. Suboptimal for sure since this breaks ftp and other things.

Anyway, I primarily just want to be able to browse SMB2/SMB3 shares and copy files between them and the memorycard and usb stick, so I just forked wLaunchELF and ripped out all the things that I think depended on running LWIP on IOP, i.e. smbman, ftp, netfs, host ... and linked with netman and lwip on EE. Then I built a thin vfs layer that would call out to libsmb2 for any path that starts with "smb2:" and it kind of works. https://github.com/sahlberg/wLaunchELF/commits/smb2

I can now browse shares on various smb2/smb3 servers and copy files to from them. I can even use SMB3 encryption, which means I can access shares on DomainControllers as well as Azure. Woohoo. This is actually all I need.

Now, I had to rip out a bunch of stuff to get this working so some things broke for sure. Mostly the reason is because I had to switch to run LWIP on EE. It should be easy, I think?, for someone that knows what they are doing and knows how to set up the stack on IOP to fix this. That way we wouldn;t need to rip out ftp, host and friends. I also broke something with loading ELF modules so my LaunchELF doesn;t actually launch any ELFs, it just reboots the PS2 when I try to :-) Well, not a problem for me, use the SMB2 variant to copy the elf from the share to the mc then have it reboot into the real wLaunchELF and then run it from the mc. Works for me.

Have a look if you want. It supports as many simultaneous shares as you want. You specify them in SMB2.CNF with blocks like this, one for each share: NAME = Home-NAS USERNAME = user PASSWORD = password URL = smb://192.168.0.1/Share/ See libsmb2 README for the URL format. You can control things like signing/encryption/version/... by the URL.

At this point, my work for my own purpose is pretty much completed. I have a filemanager that allows me to easily copy files to/from my ps2 and my SMB2 file servers. I think it should not be very hard to bring this into wLaunchELF by someone that knows these things. I think it would be very useful for a lot of users. If someone wants to give it a go and try to get it into mainline wLaunchELF I have left all the code in the smb2 branch in the repo above which should be a good starting point.

[sp193]

You could reference how LaunchELF currently works. Given what you want to do, why not replace the SMBMAN module?

Regrettably, there are no good examples around and homebrew networking has always been like battling the wilderness to me.

Basically, there are a few methods for networking:

1. LWIP on the EE, with NETMAN to manage SMAP.
2. LWIP on the IOP, with NETMAN to manage SMAP.
3. LWIP on the IOP, directly coupled with SMAP.

(3) is the legacy method, which involves ps2ip from the PS2SDK and the SMAP module from the PS2ETH project. (2) will involve ps2ip-nm, smap and netman IOP modules from the PS2SDK itself. (1) will involve the EE tcpip and netman libraries, and the smap and netman IOP modules. All of which are from the PS2SDK.

The NETwork MANager (NETMAN) is a manager for the network interface, which provides a layer of abstraction between LWIP and the network interface. It also provides API for managing the network interface, such as getting its status or overriding its operation. Since it provides some abstraction, it also allows LWIP to be moved off the 36.864MHz IOP and onto the 295MHz EE - thus improving performance. The cost, would be the complexity and that the EE will then be bothered with I/O.

You cannot simply just move drivers to the EE as the Input/Output Processor (IOP). I/O was intended to take place there, hence device drivers would run from there. Things like the I/O Manager (IOMAN) only run on the IOP, which provides access to devices via the I/O functions.

Let's quickly run through how LaunchELF works. I believe it is still using the legacy method. From here where SMBMAN and its dependencies are loaded, when required.

1. In setupPowerOff(): POWEROFF is loaded, since our CDVDMAN is old and there is no support for the power-off logic. After it is loaded, the DEV9 module is loaded, to enable access to the DEV9 interface.
2. In load_ps2ip(): PS2IP is loaded, before SMAP. This is because the SMAP module would register itself directly with LWIP.
3. Finally, SMBMAN is loaded.

SMBMAN provides access to a SMB share, via the smb: device.

I think LaunchELF's SMB support was something like a WIP. But I have never used it before, so I cannot be certain. It seems that way as I vaguely remember that it wasn't a 100% complete feature. Its connection seems to be initialized when you access the debug information screen: https://github.com/ps2homebrew/wLaunchELF/blob/master/main.c#L796

Although when completed, perhaps it can be done when the user accesses the smb: device from the File Manager. The logic would perhaps go into genFixPath(), as with the logic used for devices that deal with a filesystem. Disconnection could be added into unmountAll().

Good luck!

[sahlberg]

You could reference how LaunchELF currently works. Given what you want to do, why not replace the SMBMAN module?

Regrettably, there are no good examples around and homebrew networking has always been like battling the wilderness to me.

Basically, there are a few methods for networking:

1. LWIP on the EE, with NETMAN to manage SMAP.

2. LWIP on the IOP, with NETMAN to manage SMAP.

3. LWIP on the IOP, directly coupled with SMAP.

(3) is the legacy method, which involves ps2ip from the PS2SDK and the SMAP module from the PS2ETH project.

The NETwork MANager (NETMAN) is a manager for the network interface, which provides a layer of abstraction between LWIP and the network interface. It also provides API for managing the network interface, such as getting its status or overriding its operation. Since it provides some abstraction, it also allows LWIP to be moved off the 36.864MHz IOP and onto the 295MHz EE - thus improving performance. The cost, would be the complexity and that the EE will then be bothered with I/O.

You cannot simply just move drivers to the EE as the Input/Output Processor (IOP). I/O was intended to take place there, hence device drivers would run from there. Things like the I/O Manager (IOMAN) only run on the IOP, which provides access to devices via the I/O functions.

Let's quickly run through how LaunchELF works. I believe it is still using the legacy method. From here where SMBMAN and its dependencies are loaded, when required.

1. In setupPowerOff(): POWEROFF is loaded, since our CDVDMAN is old and there is no support for the power-off logic. After it is loaded, the DEV9 module is loaded, to enable access to the DEV9 interface.

2. In load_ps2ip(): PS2IP is loaded, before SMAP. This is because the SMAP module would register itself directly with LWIP.

3. Finally, SMBMAN is loaded.

SMBMAN provides access to a SMB share, via the smb: device.

I think LaunchELF's SMB support was something like a WIP. But I have never used it before, so I cannot be certain. It seems that way as I vaguely remember that it wasn't a 100% complete feature. Its connection seems to be initialized when you access the debug information screen: https://github.com/ps2homebrew/wLaunchELF/blob/master/main.c#L796

Although when completed, perhaps it can be done when the user accesses the smb: device from the File Manager. The logic would perhaps go into genFixPath(), as with the logic used for devices that deal with a filesystem. Disconnection could be added into unmountAll().

Good luck!

Thanks, but I know.

I did this option: "LWIP on the EE, with NETMAN to manage SMAP." It works really well. Except I had to axe a bunch of functionality that depended on LWIP on the IOP.

The struggle with IOP LWIP stack is that I can not even get it into a state where I can ping the ps2 from a remote host. And I can honestly not find any simple example applications that just do "load drivers, set ip addredd and make the host pingable." I tried but at some stage you just have to realize when you are out of depth.

I have versions of libsmb2 that compile for IOP, as well as versions of libsmb2 that compiles to EE but links against the LWIP on IOP stack. But since I can't even get the LWIP on IOP stack to initialize and become pingable ...

Anyway. My current version of LaunchELF DOES work and works VERY nicely. Its filemanager can authenticate to all SMB2/3 servers, even Azure Cloud, and I can copy files back and forth with decently good performance. (for sure, Azure with AES128CCM will never be "fast" on a cpu this slow, but it works, that is the important part.) So, the filemanager works pretty great with SMB2/3 share. Though, to get it to work, since I could not find any way to get LWIP-IOP stack to work at all, I just had to brutalize wLaunchELF and get rid of IOP-LWIP and everything that depends on it and replace it with EE-LWIP.

It works for me. I have a filemanager for PS2 that can access SMB2/3 shares and that is all I need. I tried IOP-LWIP but got nowhere so that part I will delegate to someone that knows hos this stuff works.

Please try it. Compile it and test it. It works quite well for browsing and copying files to/from smb2/3 share.

[sp193]

Yeah, which is why I am trying to help you understand where this is done in LaunchELF. This is certainly no small project. I'm actually more surprised that you got it working with the EE LWIP, since that one involves more components. Even components that LaunchELF doesn't already use.

Unlike modern consoles, network support in the PS2 is only initialized when the software does it. Hence if LaunchELF will only load the IOP modules when a specific action is done, no pinging can be done until then. For the classic procedure, network configuration is passed to the SMAP module as arguments. The IP address cannot be changed, once SMAP is loaded.

I won't be able to help you develop this project as I have stopped working on PS2 homebrew development for a few years already. At least, not until quite some time later...

[J013k]

@sahlberg have you got any test build? If yes, can you uploaded it?

[sahlberg]

J013K: I can send you a BOOT.ELF I have built with smb2 support. Send me an email at ronniesahlberg@gmail.com and I will respond to it with BOOT.ELF attached.

[TnA-Plastic]

That would be an awesome feature and I hope that parts of it could be useful in other projects like OPL.

[JacekHelka]

I did a test, the connection to NAS (SMB) - it works great! You can copy files from PS2 to and from NAS. Is it possible to add this functionality to the official version of wLaunchELF? This option makes the PS2 independent of a USB stick.

[AKuHAK]

@sahlberg I realized that you removed host: and FTP: support. While I agree that SMB: can completely replace host: (as it has almost the same approach), but I would like to keep FTP access. Personally, I am using FTP more widely than reversed operation (some limited remote control). Is it possible to launch SMB server on PS2 with the use of your library? If no, I would like to keep FTP access (it even if it is buggy). In fact, I don't see that FTP cannot exist at the same time as SMB. It is also useful to check language files. And yes, Elf loading may be tricky, but this minor bug.

[sahlberg]

@sahlberg I realized that you removed host: and FTP: support. While I agree that SMB: can completely replace host: (as it has almost the same approach), but I would like to keep FTP access.

Yes, I did because I had to move LWIP from IOP onto EE to get SMB2 to work, And I think the FTP stuff relies on LWIP running on IOP? I could be wrong. I have never done any other PS2 development before this and did not really know what I was doing.

I did have a version I tried to get to work with libsmb2 on EE and still using LWIP on IOP, thus with ftp, host and everything else still working, but I never got very far with it. I did connect to the server, authenticated and connected to the share and it worked very briefly, like listing the content of root or so before it would hang.

I would basically as far as I could tell die inside malloc() Which sounds like memory corruption of the malloc headers/lists. I did this debugging by adding simple tracing with "write loge messages to a network socket and view them with netcap." because I have no better debugging tools. At that point I just gave up and left the current LWIP on EE and thus kill off a lot of innocent other features, but hey SMB2 file manager works and that is good enough for me.

I think it should be possible to use the smb2 support with LWIP running on IOP but I do not have the skills to debug it or get it to work. So , sorry.

If it turns out I cut too deep when moving LWIP from IOP to EE and FTP should still be possible. Well. Sorry about that. I don't really know how to do it better than this. You want to do this the proper way? I would be delighted and will help as much as I can from the libsmb2 side of things.

As for using libsmb2 as a server. No, that is not possible. A server is much more complex to write than a client if not for the fact that a client can decide just how much of a protocol it wants to use and thus implement while a server will usually need to implement almost everything.

[AKuHAK]

@sahlberg Thank you for so detailed explanation. I didn't realize that now LWIP was moved into EE. You are probably right and the FTP library cannot be run on the EE, this needs more testing. Or maybe there are some design bugs in the FTP library (like initializing too much). As for debugging - it is always very difficult to debug network stuff, cause almost all tools can debug only over the network, which is not possible for testing network features. OPL debugging was made through Wireshark packet capturing (when something network side is testing). I can try to look deeper (when I get a little more free time).

[TnA-Plastic]

Btw.: Would you mind sharing the libsmb2-port "as is", maybe as a PR for the PS2SDK?

I am sure this could be useful elsewhere! I'd be interested how the performance and IOP-Load is in-game in OPL!

[AKuHAK]

libsmb2 already is in a good state. It is an external project which is actively developed and has multi-platfrom support. There is no reason to move it into the ps2 area. It can be easily added to the GitHub workflows. The only thing I can advice - add it as a submodule into ps2dk-ports but it is still not the best solution (as it is library, not port).

[sahlberg]

So, libsmb2 by default compiles against LWIP running on EE.

I did some attempts to get it to run with linking against LWIP running on IOP but never got very far debugging why it would stop working shortly after connecting to a share. See above.

If anyone wants to have a look, the wip branch for this is: https://github.com/sahlberg/libsmb2/commits/ps2ips and the main part of getting libsmb2 to link with ps2ips is this commit: https://github.com/sahlberg/libsmb2/commit/16c0a8c5231b3a887570d5ce819e38f1fb62fe66

[Apocalypse612]

This is desperately needed. I wish you all the best in getting Samba 2 working, because PS2Net (FTP), is absolutely terrible and only usable for single file transfers. There's no good/functional File Browser for PS2 HDD support on the PC either, which leaves only USB+LaunchElf for reliability. Very frustrating. The OG Xbox FTP's like a boss, it makes me sad. The PS2 needs some dev luvins.

[sahlberg]

I have created a smb2man irx that builds with the latest version of libsmb2. https://github.com/sahlberg/ps2sdk/tree/smb2man

This irx works with opl, in the sense I have a modified version of opl that uses this irx instead of smbman.irx and with it I can connect to smb2 shares and browse the list of games. It does not actually launch any games because I suspect that the irx is way too big, but connecting to the share and browsing the games in opl at least shows that smb2man.irx works in some capacity.

If you try to integrate this irx with wlaunchelf, please use the same configuration file and syntax as I use in my wlaunchelf branch for smb2 (which runs libsmb2 on EE and not as an irx). Have fun and try to do something useful with this module.

I will use this irx in some other project I am working on, but as it builds as an irx, and it kind of works with opl, it shouldn't be too hard for someone to integrate this into wLe proper.

[Apocalypse612]

Cool, thanks! Can it be used to transfer files to or from the hard drive?

[sahlberg]

Cool, thanks! Can it be used to transfer files to or from the hard drive?

Which? You mean my wlaunchelf branch (which has host/ftp/... removed) that uses libsmb2 running on EE and is not using this IRX I mentioned above? That version of wlaunchelf SHOULD work with hdd. But I don;t know, the only PS2's I have with network interfaces are slim ones and they don't have a hdd. It should work, but I don;t know.

[Haker120]

Well, digging this topic off. :D From my perspective it would be great to have SMB support not only to transfer data but to skip using USB ports to run, for example, POPStarter shortcuts having configured nano router. Now I have to run these from USB because wLE doesn't see it and when testing fork I had some otther issues, not remember what issues tho. :)

[sahlberg]

Well, digging this topic off. :D From my perspective it would be great to have SMB support not only to transfer data but to skip using USB ports to run, for example, POPStarter shortcuts having configured nano router. Now I have to run these from USB because wLE doesn't see it and when testing fork I had some otther issues, not remember what issues tho. :)

It should be possible. But I do not have the skills or knowledge of PS2 network and filesystem stack to do it. For now you can use my fork of wlaunchelf: https://www.psx-place.com/threads/smblaunchelf.32434/page-2 It supports SMB2/3 as well as NFSv3/v4 but as it moves LWIP from IOP to EE it has to sacrifice some other features, like FTP. So unlikely to be possible to backport into upstream.

Hey, it is open source. The only thing we need is someone knowing what they are doing porting this all from EE to IOP and then mergig into upstream wle. :-)

[rickgaiser]

@sahlberg I see you have many different network communication libraries. Both file and block based. Have you ever thought about an AoE (ATA over Ethernet) library?

I think that protocol has great potential for the ps2, both as server (access ps2 block devices from pc), and as client (access pc block device from ps2). Not needing a TCP/IP stack is a big benefit for the IOP. It could replace both NBD and UDPBD on the ps2, becouse it has the potential to be both standard (like NBD) and fast (like UDPBD).

[sahlberg]

I did look at AoE about 20 years ago. I even wrote the wireshark dissector for it :-) It is a nice little protocol, and simple. But I don't think it ever got very popular.

Adding a block device today I think using iscsi would be better. It is a much more complex protocol but it is widely available. libiscsi might be a bit too big and bloated for the ps2 but it should be feasible to strip it down into a minimal fork. Getting rid of RDMA, authentication, crypto, and culling the SCSI opcodes that are available should be possible to get it very very small. Possibly the only opcodes one would need would be Inquiry (to find the size) and read10 (to read the data) Would it be as small as NBD? probably not but it would be iscsi and iscsi targets are available everywhere.

Anyway, maybe I should write a aoe client library. It could be fun.

[sahlberg]

For a replacement for UDPBD I would probably design a minimalist read-only network fs from scratch. Maybe something like this:
It runs on UDP and is stateless, i.e. no open()/close() on the wire. Single threaded and synchronous so no need to do any request/response matching. It supports three operations : read, stat and readdir.

The packets could be encoded like:

Read request: 4 bytes: command == 0 (0==read, 1==stat, 2==readdir) 4 bytes: length 8 bytes: file offset 4 bytes: full-path-file-name-length n bytes: full-path-filename

Read response: 4 bytes: errno, 0==success 4 bytes: number of bytes read n bytes: data

And similar for stat and readdir. Minimal read-only filesystem. No authentication, no access control. A server could be written in a few hours in python and be fast enough for the ps2. A client library on the iop should weight in at under a kilobyte, plus 64kb for buffer for udp packet and 64kb for work.

[rickgaiser]

ISCSI and NBD are both well supported protocols, but they both use TCP/IP. And using TCP/IP is too much for the IOP (MIPS R3000 / Playstation1 CPU / slow) to utilize the network at 100Mbps. So I've been looking into UDP alternatives. The simple protocol I created (UDPBD) has some special alignments that allow the ps2 to use DMA without the need for intermediate buffers.

64k+64k on the IOP is a NO-GO. With every KiB used, a game that needs a lot of IOP memory stops working.

If there's no standard protocol than a custom protocol is the only way. However I think stateless (sending the entire path to a file each command) will be too simple and slow. At least open/close of a file would be needed. Perhaps the IOP can run only by using (int32_t) file descriptors, after the loader has opened the needed files (like ISO / VMC). Reading and writing to these file descriptors can then use the same protocol as UDPBD does. Having the same speed, but file based instead of block based.

[sahlberg]

Ok, that sounds reasonable. So you will only read one 2048 byte sector at a time? IF memory is that critical I would then suggest you only read 1024 bytes at a time. The MTU on the 100baseT going to be 1500. Thus if you request a UDP packet with a 2kb payload this will be fragmented into one ~1500byte frame and one ~600byte frame and reassembled inside LWIP. IF you have a modest amount of packetloss and the ~600byte fragment is lost then you will end up with the ~1500byte fragment still sitting inside LWIP consuming memory until whatever fragment reassembly timeout expires. If you drop the read size to 1024 then the whole response will fit in a single frame and no reassembly needed.

That should not affect latency I think since you could just send the two 1024byte reads back to back concurrently.

[Wolf3s]

Well, digging this topic off. :D From my perspective it would be great to have SMB support not only to transfer data but to skip using USB ports to run, for example, POPStarter shortcuts having configured nano router. Now I have to run these from USB because wLE doesn't see it and when testing fork I had some otther issues, not remember what issues tho. :)

It should be possible. But I do not have the skills or knowledge of PS2 network and filesystem stack to do it. For now you can use my fork of wlaunchelf: https://www.psx-place.com/threads/smblaunchelf.32434/page-2 It supports SMB2/3 as well as NFSv3/v4 but as it moves LWIP from IOP to EE it has to sacrifice some other features, like FTP. So unlikely to be possible to backport into upstream.

Hey, it is open source. The only thing we need is someone knowing what they are doing porting this all from EE to IOP and then mergig into upstream wle. :-)

Updated fork: https://github.com/Wolf3s/wLaunchELF.git