[FR] Migrate to Smbv2 or Smbv3 [SMB]

[beatstick]

The server part seems to rely on Smbv1. I cannot connect the ps2 to a server running win7 or libreelec. Smbv1 is disabled on both systems, as it is deemed insecure nowadays. (Vulnerable to Ransomware). Would you consider migrating to a more secure version of Samba? Otherwise a hint inside the readme would be great that Smbv1 needs to be activated to run opl and maybe that the server should not be connected to the internet for security reasons.

[sp193]

This is related to the PS2SDK as well, FYI. Both will have to be upgraded, for the transition to a new SMB standard to be complete.

[TnA-Plastic]

@beatstick (Whuuuaaat? What a name! :'-D):

Technically it is quite possible...

However, SMB-Support is a BIT 'hackish' since it's introduction into 'USBLD' (thus making it the first OPL)... We are 'driving' the IOP like a race-car (or Koenigsegg Agera, or a rocket-center) while it actually is a 'Lada' or a 'Trabant'...

I am not sure, if SMBv2/3 would introduce a lot overhead on the IOP's RAM and/or CPU-Usage, which is my main concern (especially due to game-compatibility)... :-/

[yoshi314]

I wonder if nfs or plain http would make more sense. The latter might be problematic wrt VMC, but it could be pretty simple.

[sp193]

I have started work on one, since it doesn't seem like anybody is. It will be done when I can.

Please refer to https://github.com/ps2dev/ps2sdk/issues/72 for any progress updates. I plan to update the driver within the PS2SDK first, before adding a micro-driver here. The SMBMAN module within the PS2SDK will support the SMB 2 protocol, with support for SMB 1.0 for backward-compatibility. OPL will load the appropriate micro-driver, based on the selected protocol.

[B17C0D3]

is nfs not an option?

[sahlberg]

If people are still working on this, please see https://github.com/sahlberg/libsmb2

It is a very portable high performance, small footprint, smb2/3 client that builds on almost anything. It even runs on ESP32, which is a microcontroller with 520kb of RAM, running a tiny real time os.

It connects and interoperates with samba, windows, azure, osx, and pretty much any smb2/3 capable server.

The size of the code for the full blown library is ~120kb, but can be stripped down to ~60-70kb by removing features such as encryption, dcerpc (for share enumeration) and other non-critical features. If anyone wants to give it a try, I am happy to provide any and all support I can from the libsmb2 side to help you.

[rickgaiser]

Hi and welcome! It sounds like a great alternative/improvement to the SMB implementation we have right now. I'm also a big fan of using standard software instead of ps2 specific software, becouse it makes maintaining things more easy on the ps2 side.

If it runs on an ESP32, then maybe it can run on the ps2. However, the network adapter and tcp/ip stack are not located on the 300MHz EE CPU, but on the 33MHz IOP CPU. I'm not sure how familiar you are with the ps2, but the IOP (I/O Processor) is the old 33MHz MIPS R3000 Playstation1 CPU. It's got 2MiB of ram total, to do all IO processing. Like controllers, memorycard, sound, usb, etc... It runs something similar to a tiny real time os.

We use LwIP as our tcp/ip stack. On the other side of the library we need to interface to the IOP file system, using the iop_device_ops_t structure: https://github.com/ps2dev/ps2sdk/blob/34b6875d632cf171239624c7acff1452f92792be/iop/kernel/include/ioman.h#L69

So to make it work we need something like this:

Do you know if libsmb2 works together with LwIP? Would it be difficult to use the library from a structure like the iop_device_ops_t?

[sahlberg]

Hi and welcome! It sounds like a great alternative/improvement to the SMB implementation we have right now. I'm also a big fan of using standard software instead of ps2 specific software, becouse it makes maintaining things more easy on the ps2 side.

If it runs on an ESP32, then maybe it can run on the ps2. However, the network adapter and tcp/ip stack are not located on the 300MHz EE CPU, but on the 33MHz IOP CPU. I'm not sure how familiar you are with the ps2, but the IOP (I/O Processor) is the old 33MHz MIPS R3000 Playstation1 CPU. It's got 2MiB of ram total, to do all IO processing. Like controllers, memorycard, sound, usb, etc... It runs something similar to a tiny real time os.

We use LwIP as our tcp/ip stack. On the other side of the library we need to interface to the IOP file system, using the iop_device_ops_t structure: https://github.com/ps2dev/ps2sdk/blob/34b6875d632cf171239624c7acff1452f92792be/iop/kernel/include/ioman.h#L69

So to make it work we need something like this:

Do you know if libsmb2 works together with LwIP? Would it be difficult to use the library from a structure like the iop_device_ops_t?

I had a look at LwIP and it does support an optional Berkley socket layer so that should be all we need. Without looking too deeply into LwIP right now, as it is a very small and limited stack, I see only 3 potential issues. All three would be easy to handle on libsmb2 side.

1, non-blocking socket support. libsmb2 should work even is LwIP does not support this but I have not tested it in a while. Would be a bug and easy for me to fix if it doesn't work. The only drawback if LwIP does not support non-blocking sockets is that "waiting for a response from the server" essentially becomes a busy-wait loop, so it wastes CPU while you have requests in flight. It works but wastes cpu.

2, Support for getaddrinfo() to resolve names. If you don't have getaddrinfo() we would need to write a small replacement function. This is similar to how ESP32 SDK does not have readv//writev support so we had to add simple replacement functions for it. See here as example: https://github.com/sahlberg/libsmb2/blob/master/lib/compat.c

3, IPv6. Libsmb2 currently assumes that all platforms have ipv6 support. If LwIP does not have IPv6 support we will need to add some #ifndef PS2 conditionals to 3-4 places in libsmb2:lib/socket.c to remove the dependency of it.

Memory and CPU requirements. 33MHz should be fine. It is not going to be super-fast but libsmb2 should be faster than the current SMB1 implementation (which I assume is based on old version of libsmbclient or the old smb1 implementation in vlc). Main reason for that is because libsmb2 does zero-copy for read/write data. I.e. it does not do any memcpy of data between the application and temporary buffers (except if you use encryption) so data is read/written directly between the application memory to/from the socket. Memory, you have 2MB RAM which is 4 times as much what ESP32 has. That is more than enough. What we might need to do is add an #ifdef to clamp the maximum PDU size inside libsmb2. It defaults to 64kb which might be a bit much depending on the budget for how much RAM we can use. On ESP32 we clamp this to just 512 bytes instead to save some memory https://github.com/sahlberg/libsmb2/blob/9268322f177309b282fc3f9e0d06a5960b73fc8c/lib/libsmb2.c#L81 The drawback is that the smaller you make the maximum read/write size the slower things will get since you read/write in smaller chunks. Since this would mostly read/write ISO images, maybe clamping it to 2048 bytes to match the sector size would be a good idea.

iop_device_ops_t I had a look at this structure and it should be easy to interface libsmb2 to it. Some of the methods, like format and ioctl, would not make sense on a network filesystem, but all other functions are available in libsmb2 and they do come with a posix-like API that is almost identical to normal posix functions. So we have for example smb2_open() that is modelled after the normal open() call : https://github.com/sahlberg/libsmb2/blob/9268322f177309b282fc3f9e0d06a5960b73fc8c/include/smb2/libsmb2.h#L476

So as long as you have a budget for ~100-150kb for the object code (we can strip this down) and a few tens of kb for RAM it should be fine. Have a look at libsmb2 and in particular look at include/smb2/libsmb2.h to see the posix-like API, smb2_open and friends. Also "git grep ESP_PLATFORM" to see the modifications we needed for ESP. We will likely need some minor tweaks for PS2 as well.

I am more than happy to work with you and do all the modifications and fix all the issues you encounter to get this working on PS2. This would be awesome!

[rickgaiser]

1, non-blocking socket support. libsmb2 should work even is LwIP does not support this but I have not tested it in a while. Would be a bug and easy for me to fix if it doesn't work. The only drawback if LwIP does not support non-blocking sockets is that "waiting for a response from the server" essentially becomes a busy-wait loop, so it wastes CPU while you have requests in flight. It works but wastes cpu.

Why would a blocking call waste CPU cycles? Isn't that depending on how LwIP blocks? Busy waiting is a bad idea on the IOP, becouse it would block other IO from being processed. Like sound or the controller input. The lower level LwIP "RAW" API does provide TCP data reception is callback based - an application specified callback function is called when new data arrives. Perhaps an #ifdef LWIP would possible? The IOP also has threads, mutexes, semaphores and events if that makes any difference.

2, Support for getaddrinfo() to resolve names. If you don't have getaddrinfo() we would need to write a small replacement function.

Yes, an #ifdef LWIP in compat.c I guess.

3, IPv6. Libsmb2 currently assumes that all platforms have ipv6 support. If LwIP does not have IPv6 support we will need to add some #ifndef PS2 conditionals to 3-4 places in libsmb2:lib/socket.c to remove the dependency of it.

LwIP does have IPv6 support, but that does not mean we use it or need it. I would choose something like #ifdef USE_IPV6 for these. So both LwIP and libsbm2 can be slimmed down, or IPv6 added when needed.

Since this would mostly read/write ISO images, maybe clamping it to 2048 bytes to match the sector size would be a good idea.

Yes, sounds like a good idea.

So as long as you have a budget for ~100-150kb for the object code (we can strip this down) and a few tens of kb for RAM it should be fine. Have a look at libsmb2 and in particular look at include/smb2/libsmb2.h to see the posix-like API, smb2_open and friends.

Great! That does look simple to implement from the posix side. We do have ~100-150kb, but there's a catch. If writing our own programs we control the budget, and we can use whatever memory we like. However, when emulating a game from DVD, the game controls the budget, and we have 'hide' our emulation in whatever memory is left. So the more memory we use, the less compatibility will be.

I am more than happy to work with you and do all the modifications and fix all the issues you encounter to get this working on PS2. This would be awesome!

I would love to work on this but unfortunately I do now have the time to take up another project. Besides work and life, I'm also working on the software for an SD adapter (search ps2 sio2sd), and I'm trying to upgrade GCC to the latest version (we're still using gcc 3.2.3). If you can make libsmb2 compatible with LwIP, then it shouldn't be difficult to make it work on the ps2 for normal homebrew. However, OPL-ingame, is a lot more difficult to get working, and for now I don't think I'll have time for that. So as far as time goes, I can most likely make it show a list of games in OPL, but I cannot make the games run ;). But perhaps @sp193 is up for a new challenge, what do you think?

[sahlberg]

Why would a blocking call waste CPU cycles? Isn't that depending on how LwIP blocks? Busy waiting is a bad idea on the IOP, becouse it would block other IO from being processed. Like sound or the controller input.

I misremembered. Sorry. If LwIP does not support non-blocking operations, worst thing that will happen is that the the thread will block until data becomes available. That would mostly/only affect single-threaded event-drive applications. There is an ioctl for FIONREAD that could be used, but it is 1, not portable, and 2, on some platforms where it is supported it is broken :-(

So as long as you have a budget for ~100-150kb for the object code (we can strip this down) and a few tens of kb for RAM it should be fine. Have a look at libsmb2 and in particular look at include/smb2/libsmb2.h to see the posix-like API, smb2_open and friends.

Great! That does look simple to implement from the posix side. We do have ~100-150kb, but there's a catch. If writing our own programs we control the budget, and we can use whatever memory we like. However, when emulating a game from DVD, the game controls the budget, and we have 'hide' our emulation in whatever memory is left. So the more memory we use, the less compatibility will be.

Ok. Getting the object size down to ~90kb would be almost trivial by just getting rid of encryption, signing and dcerpc (used for share enumeration). This I can make possible with almost no effort by just adding a few #ifdefs and doocument them. I will do that regardless of if you want to work on this. I have planned to modularize the featureset to slim down the footprint to handle even smaller and more constrained embedded devices.

Getting from ~90kb to ~70 is possible with almost no effort too by shedding the authentication layer. That would mean only "guest" access to the share will work. No username/password. That would also be almost no effort required.

That would bring the code size to ~70kb and then the run-time memory requirement to ~3-4kb.

It is possible to get even smaller but by now all low hanging fruit is gone so everything below ~70kb would become significantly more effort. Not impossible, just a lot more work. And if we need to, I will do it.

I am more than happy to work with you and do all the modifications and fix all the issues you encounter to get this working on PS2. This would be awesome!

I would love to work on this but unfortunately I do now have the time to take up another project. Besides work and life, I'm also working on the software for an SD adapter (search ps2 sio2sd), and I'm trying to upgrade GCC to the latest version (we're still using gcc 3.2.3). If you can make libsmb2 compatible with LwIP, then it shouldn't be difficult to make it work on the ps2 for normal homebrew. However, OPL-ingame, is a lot more difficult to get working, and for now I don't think I'll have time for that. So as far as time goes, I can most likely make it show a list of games in OPL, but I cannot make the games run ;). But perhaps @sp193 is up for a new challenge, what do you think?

I understand the scarcity of spare time for projects well. @sp193 you want to try this? I will provide all help you need from the libsmb2 side to make sure it will work for you. But I am completely ignorant about how the PS2 coding works.

Maybe an alternative would be to get libsmb2 to compile and work in PS2/LwIP would be to get some homebrew to use it first. Do any of you know of any actively developed homebrew where it would make sense to add SMB2/3 support and I can try to approach them. After looking more into LwIP and PS2. I think the majority of the work to get libsmb2 running would mostly be tweaking which header files should be included and ifdef out the ones that are not available on PS2. Second would be to conditionally add replacements for the posix functions that may be missing. (like readv/writev/...) but that should not be hard. As long as I know which functions are missing, i.e. someone tells me what caused the linker to fail, and then I can easily add simple replacements for anything that is missing.

[sp193]

No thanks. I would rather not spend time on PS2 projects these days.

Our existing code might be original, tailored for the PS2. In-game OPL has a heavily customized version, which had the authentication part split off into a non-resident module - all done in the name of reducing footprint.

[sahlberg]

No thanks. I would rather not spend time on PS2 projects these days.

Our existing code might be original, tailored for the PS2. In-game OPL has a heavily customized version, which had the authentication part split off into a non-resident module - all done in the name of reducing footprint.

Fair enough. Just out of curiosity. How small footprint do you think it needs to have to be viable for OPL? I am now stripped down to ~35kb code size which includes authentication and think I might get to just under 30kb before it will start taking a lot more effort to go smaller.

[sp193]

Back in my day, we were supposed to be always short of memory. I don't know about now, but I suppose you got to find the OPL developers and ask them about it... Sometimes it is just about trial-and-error. Usually, a game is designed and developed against a specific SDK, and is free to use the remaining memory as desired. But here, we're trying to write modules that are compatible with all games.

For the ultimate compatibility, I believe the memory consumed by the new IOP modules must not exceed the memory consumed by the original modules.

[sahlberg]

I have created two separate branches of libsmb2 if someone wants to give it a try in the future.

The "small" branch which is a stripped down, read-only, but mostly complete version of libsmb2 that can be used to authenticate, scan directories and open a file handle. This compiles down into ~35kb.

And a "super-small" branch that compiles down into ~4kb. This branch ONLY implements smb2_read() and smb2_read_async() but will accept the context and handle that were opened from the "small" module.

So a potential flow could be: 1, dynamically load the 36kb "small" module. Authenticate, connect to the share, scan directories and eventually open a file for reading. 2, unload this module 3, load the 4kb "super-small" module and use the smb2_read() / smb2_read_async() methods using the context and handle from step 1.

This would allow you to reading from a file using only a 4kb foot-print for the smb2 module.

This was fun, and useful. In particular the "small" branch will become useful for me and I will merge parts of it mack into mainline since it will allow constrained devices to optionally shed un-needed functionality.

If someone wants to try add smb2 support to OPL in the future, I will leave the "small" and "super-small" branches on github.

[rickgaiser]

Thank you, I can't promise anything but I would like to try in the future.

So a potential flow could be: 1, dynamically load the 36kb "small" module. Authenticate, connect to the share, scan directories and eventually open a file for reading. 2, unload this module 3, load the 4kb "super-small" module and use the smb2_read() / smb2_read_async() methods using the context and handle from step 1.

I think this could work for OPL. The GUI would use the "small" module and the "ingame" could use the "super-small" version. If we separate like this the "small" version could even be a lot larger, as long as the "super-small" version stays as small as possible ;).

[sahlberg]

Thank you, I can't promise anything but I would like to try in the future.

So a potential flow could be: 1, dynamically load the 36kb "small" module. Authenticate, connect to the share, scan directories and eventually open a file for reading. 2, unload this module 3, load the 4kb "super-small" module and use the smb2_read() / smb2_read_async() methods using the context and handle from step 1.

I think this could work for OPL. The GUI would use the "small" module and the "ingame" could use the "super-small" version. If we separate like this the "small" version could even be a lot larger, as long as the "super-small" version stays as small as possible ;).

Cool. I have updated the small and super-small branches with a patch that makes it compile (and work) for building against ps2sdk. I have also a branch ps2 which is the full blown master branch with this patch applied too. This patch will go away at some stage when I merge the ps2 ee support into the main master branch. But the small and super-small branches will remain.

I tested this by creating a test program based on the example in "samples/network/tcpip-basic/" in the sdk and added calls to libsmb2 to connect, authenticate, and map a SMB2/3 share from the network. I ran the resulting elf binary from a ps2 slim and it worked. Wooohooo. But the heavy lifting should be done now if someone wants to try to integrate this into opl.

To compile : cd libsmb2/lib make -f Makefile.PS2_EE

[rickgaiser]

I'm sorry to disappoint you, but you've created the driver on the wrong (easy to program) cpu. Porting software to the EE has become fairly easy recently. This does work for the OPL GUI, where we have full control over the EE. But this does not work when emulating the DVD drive for games.

This, unfortunately, requires the drivers to be on the IOP side. The network stack has become very flexible in the sense that it can be loaded on the EE or the IOP. To ilustrate, here's an image I created a few years ago: Using option2 it should be possible to create the smb implementation on the IOP side. I've actually tried compiling libsmb2 a few days ago (using the iop compiler 'iop-gcc'), and ran into all sorts of issues with header files not existing, like "stdint.h".

I created the following Makefile in the lib folder:

IOP_BIN  = libsmb2.irx
IOP_OBJS = aes.o \
            aes128ccm.o \
            alloc.o \
            dcerpc.o \
            dcerpc-lsa.o \
            dcerpc-srvsvc.o \
            errors.o \
            init.o \
            hmac.o \
            hmac-md5.o \
            krb5-wrapper.o \
            libsmb2.o \
            md4c.o \
            md5.o \
            ntlmssp.o \
            pdu.o \
            sha1.o \
            sha224-256.o \
            sha384-512.o \
            smb2-cmd-close.o \
            smb2-cmd-create.o \
            smb2-cmd-echo.o \
            smb2-cmd-error.o \
            smb2-cmd-flush.o \
            smb2-cmd-ioctl.o \
            smb2-cmd-logoff.o \
            smb2-cmd-negotiate.o \
            smb2-cmd-query-directory.o \
            smb2-cmd-query-info.o \
            smb2-cmd-read.o \
            smb2-cmd-session-setup.o \
            smb2-cmd-set-info.o \
            smb2-cmd-tree-connect.o \
            smb2-cmd-tree-disconnect.o \
            smb2-cmd-write.o \
            smb2-data-file-info.o \
            smb2-data-filesystem-info.o \
            smb2-data-security-descriptor.o \
            smb2-data-reparse-point.o \
            smb2-share-enum.o \
            smb3-seal.o \
            smb2-signing.o \
            socket.o \
            sync.o \
            timestamps.o \
            unicode.o \
            usha.o

IOP_CFLAGS += -Wall -O2 -I. -I../include -I../include/smb2

all: $(IOP_BIN)

clean:
        rm -f -r $(IOP_OBJS) $(IOP_BIN)

install: $(IOP_BIN)
        cp $(IOP_BIN) $(PS2SDK)/iop/irx

include $(PS2SDK)/Defs.make
include $(PS2SDK)/samples/Makefile.pref
include $(PS2SDK)/samples/Makefile.iopglobal

Can you try compiling for the IOP?

PS: Congratulations are still in order I think. Even though it's on the EE, it's still a great achievement in a short time ;). Perhaps it's even usefull for other applications. uLE perhaps?

[sahlberg]

I'm sorry to disappoint you, but you've created the driver on the wrong (easy to program) cpu. Porting software to the EE has become fairly easy recently. This does work for the OPL GUI, where we have full control over the EE. But this does not work when emulating the DVD drive for games.

This, unfortunately, requires the drivers to be on the IOP side. The network stack has become very flexible in the sense that it can be loaded on the EE or the IOP. To ilustrate, here's an image I created a few years ago: Using option2 it should be possible to create the smb implementation on the IOP side. I've actually tried compiling libsmb2 a few days ago (using the iop compiler 'iop-gcc'), and ran into all sorts of issues with header files not existing, like "stdint.h".

I created the following Makefile in the lib folder:

IOP_BIN  = libsmb2.irx
IOP_OBJS = aes.o \
            aes128ccm.o \
            alloc.o \
            dcerpc.o \
            dcerpc-lsa.o \
            dcerpc-srvsvc.o \
            errors.o \
            init.o \
            hmac.o \
            hmac-md5.o \
            krb5-wrapper.o \
            libsmb2.o \
            md4c.o \
            md5.o \
            ntlmssp.o \
            pdu.o \
            sha1.o \
            sha224-256.o \
            sha384-512.o \
            smb2-cmd-close.o \
            smb2-cmd-create.o \
            smb2-cmd-echo.o \
            smb2-cmd-error.o \
            smb2-cmd-flush.o \
            smb2-cmd-ioctl.o \
            smb2-cmd-logoff.o \
            smb2-cmd-negotiate.o \
            smb2-cmd-query-directory.o \
            smb2-cmd-query-info.o \
            smb2-cmd-read.o \
            smb2-cmd-session-setup.o \
            smb2-cmd-set-info.o \
            smb2-cmd-tree-connect.o \
            smb2-cmd-tree-disconnect.o \
            smb2-cmd-write.o \
            smb2-data-file-info.o \
            smb2-data-filesystem-info.o \
            smb2-data-security-descriptor.o \
            smb2-data-reparse-point.o \
            smb2-share-enum.o \
            smb3-seal.o \
            smb2-signing.o \
            socket.o \
            sync.o \
            timestamps.o \
            unicode.o \
            usha.o

IOP_CFLAGS += -Wall -O2 -I. -I../include -I../include/smb2

all: $(IOP_BIN)

clean:
        rm -f -r $(IOP_OBJS) $(IOP_BIN)

install: $(IOP_BIN)
        cp $(IOP_BIN) $(PS2SDK)/iop/irx

include $(PS2SDK)/Defs.make
include $(PS2SDK)/samples/Makefile.pref
include $(PS2SDK)/samples/Makefile.iopglobal

Can you try compiling for the IOP?

PS: Congratulations are still in order I think. Even though it's on the EE, it's still a great achievement in a short time ;). Perhaps it's even usefull for other applications. uLE perhaps?

Thanks. I will give a try on IOP this coming weekend. So a working solution could be the full ps2/master branch of libsmb2 running on EE for the GUI and then the ~4kb super-small branch modified to run on the IOP.

( I downloaded the SDK yesterday and installed it. It is really well documented and well maintained. It just installed like the instruactions said without any issues at all. If only all SDKs where that easy to set up and get running. Hats off to the sdk maintainers. )

[sahlberg]

Please check my ps2 branch and the super-small branch. (the small read-only branch might be redundant) It does compile the library for IOP now.

In the "ps2" branch, we have a full libsmb2, with everything and it compiles into a libsmb2.a without compiler warnings for both EE and IPL platforms. Use : $ cd lib $ make clean $ make -f Makefile.PS2_EE or $ make clean $ make -f Makefile.PS2_IOP to compile.

I have tested PS2_EE and it works with sample programs on a ps2 slim as far as my testing goes.

I have not tested PS2_IOP because I can not easily build applications for it (or even know how to do so) but the IOP builds and links to a libsmb2.a library without any compiler warnings so I assume it should mostly work. The changes in the IOP compatibility layers is all based on reading the header files in the ps2 sdk. I.e. there is no lwip_read function but there is a lwip_recv function etc etc. It compiles to libsmb.a without warnings so I assume any external symbols we need do actually exist in external libraries. If not, let me know which functions are missing and I can add them. See compat.c and compat.h for the meat of the stuff added for PS2 compatibility.

[sahlberg]

A basic example on how to authenticate and connect to a share using libsmb2:

    smb2 = smb2_init_context();
    if (smb2 == NULL) {
            scr_setXY(10, y++);
            scr_printf("init context failed.\n");
            return 0;
    }
    scr_setXY(10, y++);
    scr_printf("init context SUCCEEDED.\n");
    sleep(10);

    smb2_set_user(smb2, USER);
    smb2_set_password(smb2, PASSWORD);

    if (smb2_connect_share(smb2, SERVER, SHARE, USER) < 0) {
            scr_setXY(10, y++);
            scr_printf("smb2_connect_share failed. %s\n", smb2_get_error(smb2));
            return 0;
    }

smb2_connect_share is sync and blocking interface, use smb2_connect_share_async() to use the async event driven API.

After these calls are done and you are connected to the share you can use calls such as smb2_opendir(_async)(), smb2_open(_async)() ... see libsmb2.h for the API.

[sahlberg]

I tried to fix up the ps2 branch with the full libsmb2 to build as a irx and it almost does. It has trouble with linking with errno I haven't figured out how to fix yet. (I don't really know what I am doing honestly.)

If we can solve the issue with errno, and get it to work as an irx for IOP for the ps2 branch (full blown libsmb2) it should be easy to port the changes to the super-small branch.

[sahlberg]

Sorry for yet more spam, I managed to solve the remaining issues and it now builds into a libsmb2.irx. The super-small branch clocks in at ~21kb (I guess it contains a lot more additionall stuff than only the libsmb2 part) which is about half the size of smbman.irx that clocks in at ~46kb.

The ps2 branch, which is the full blown libsmb2 with ps2 ee/iop support is about ~180kb when built as an irx.

So, if this works, we should be able to have a system where 1, load libsmb2.irx from the ps2 branch and use it for the ui. 2, once we want to launch a game we open the file so we have a context and a filehandle 3, and then unload the ps2:libsmb2.irx without freeing the context and the filehandle 4, then load the super-small:libsmb2.irx that only contains the sync and async read and pread methods 5, but continue to use the context and handle we got from step 2

And we are now in full smb2/3 support for reading the sectors for a game with even ~25kb less RAM used than with the current smbman.irx. Victory!

[rickgaiser]

Congratulations and thank you for the 'spam' ;). I'm on vacation and reading your messages with much interest. I'll try to create a sample program with the module(s) from your branch when I get the chance.

[sahlberg]

I tried to start building it as a filesystem module with an iop_device_ops_t like smbman but I cant even get the init function to be invoked.

I tried to fixup labs/smblab as that looks like a nice example on how to use smbman running on the iop by fixing the makefile so it compiles but it doesn't work either :-( So that is sad.

The ps2 branch contains my changes to build an initial iop_device_ops_t containing only an init and deinit function. I am giving up.

[sahlberg]

Any help now is appreciated. I can't get any further right now.

[sp193]

You need to load the module, perhaps with something like LoadModule(). The right function to use, depends on what and where you are loading the module from. After it is loaded, remains resident and registers itself with IOMAN (or iomanX from the homebrew SDK), then the I/O functions can be used to do something. If you have not familarized yourself with IOP programming, please do refer to tutorials. Even though they can be very old and using archaic methods of homebrew software development, the basics remain the same (i.e. modules have export and import tables, they use static linking). Referring to the Sony documentation would probably help too, since we're using their kernel.

Homebrew software generally use the default modules that come with the PS2, which are a standard set of modules from the first SCPH-10000 to the last SCPH-90000 model made. According to what Sony wrote about the TOOL PS2's testing mode, those IOP modules are functionally equivalent to the modules from the official release 1.3.4 - which is very old (from 1999). As long as the IOP is reboot with no arguments (an empty string is passed), then the default modules are loaded.

Please note that the IOP kernel has restrictions, to prevent licensees from loading unsigned code from user-writable media. As such, (Sif)LoadModule() will never load modules from the memory card. There is sbv_patch_disable_prefix_check() to disable this check. But I don't suppose you're using this method.

LoadModuleBuffer()'s SIF RPC was not implemented as of the PS2's release, which is why it must be enabled with sbv_patch_enable_lmb(). Sony also had to do this, for the HDD Browser. Only after you execute this patch, will functions that involve SifLoadModuleBuffer() work - such as SifExecModuleBuffer().

Remember to check the return value. It should be a positive number that represents the module ID. This is a running number, so it is not supposed to be a large value. While negative numbers indicate an error, calling the SifLoadModuleBuffer() RPC before it is fixed, will cause an undefined value to be returned - which might be still a positive number.

[Migsi]

@sahlberg @rickgaiser any updates on this one? This would easyly be one of the hugest improovements over the last years.

[sahlberg]

No update and no progress.

I did spend a lot of time on this but I don't really know PS2 development. I have zero experience in PS2 homebrew development or debugging. I DID get a version of wLaunchElf to work with libsmb2 using a chainsaw approach (moving lwip from iop onto ee and chainsawing away anything and everything that depended on lwip on iop, which realistically is almost everything in wle except the file manager function.) I have tried seriously to do some work with iop but I am just out of my depths. I can not debug I can not even build smbman applications and get them to work (I build on fedora, maybe that is the issue) so I really can not contribute anymore. I CAN get libsmb2 to build as a link library using the iop build environment but that is it.

TBH. It would be very nice to have smb2 support, but really, a fat ps2 with a gamestar adapter for 30$AUD is a much better solution. https://www.ebay.com.au/itm/Expansion-Game-Interface-HDD-Adapter-Gamestar-HDD-Adaptor-For-Sony-No-Ide-2/193725095282?hash=item2d1aea5d72:g:~9kAAOSw~5Rf9l3m&frcectupt=true

Sure no SMB support but faster and cheaper and it works. I have 9 PS2, 7 of them are fat and 2 of those have a GameStar. It works and does its purpose.

[auanasgheps]

Sure no SMB support but faster and cheaper and it works.

My two cents here:

I believe SMB and HDD approach are extremely different and should not be compared. It's like going to work using a car or public transport.

HDD You need a fat PS2, which is heavy and noisy. An old HDD is extremely cheap (you may already have it) but you don't know how long is going to last, plus makes even more noise. But the biggest thing that annoys me is the procedure to load HDD with games, which is is extremely slow and tedious. No "try and play" approach otherwise you'd better give up.

SMB You can use a slim PS2, which are lighter, have less footprint and have everything they need. I personally prefer this model. The Gamestar adapter can cost like the whole console. We're talking about small figures, but still. SMB allows to "try and play" because loading content is as easy as copy-paste.

Somebody prefers HDD, other SMB, according to what they have and need. In conclusion, I appreciate that SMBv2 is feasible, although not yet implemented, is something I would heavily use.

[Migsi]

No update and no progress.

I did spend a lot of time on this but I don't really know PS2 development. I have zero experience in PS2 homebrew development or debugging. I DID get a version of wLaunchElf to work with libsmb2 using a chainsaw approach (moving lwip from iop onto ee and chainsawing away anything and everything that depended on lwip on iop, which realistically is almost everything in wle except the file manager function.) I have tried seriously to do some work with iop but I am just out of my depths. I can not debug I can not even build smbman applications and get them to work (I build on fedora, maybe that is the issue) so I really can not contribute anymore. I CAN get libsmb2 to build as a link library using the iop build environment but that is it.

TBH. It would be very nice to have smb2 support, but really, a fat ps2 with a gamestar adapter for 30$AUD is a much better solution. https://www.ebay.com.au/itm/Expansion-Game-Interface-HDD-Adapter-Gamestar-HDD-Adaptor-For-Sony-No-Ide-2/193725095282?hash=item2d1aea5d72:g:~9kAAOSw~5Rf9l3m&frcectupt=true

Sure no SMB support but faster and cheaper and it works. I have 9 PS2, 7 of them are fat and 2 of those have a GameStar. It works and does its purpose.

No worries for not making progress, you already did quite a lot to bring SMBv2/3 support forward! Its good to hear you still have this on your mind though. I was hoping for @rickgaiser of @sp193 to have a look into integrating this further, as this might be one of the most awaited features in a while (primarily because of the security implications of running a SMBv1 instance). Also as version 1.0 was officially released a few days ago, there might be more time availabe to spend on the integration task now?

Surely using an HDD might be feasible too, but I agree to @auanasgheps that especially for those owning a single slim ps2 it might not be an option.

I fully understand this whole project is driven by a few people in their free time and do not want to urge anybody to take care of this soon, but I'd already be happy to hear, that this is is at least considered as an important feature and might be worked on in the near future. :)

[mlaux]

I made a little progress on this - I created a barebones IRX that simply calls smb2_init_context and smb2_connect_share as a test. I'm running into an issue where smbv2's iop_connect() compat wrapper, which calls lwip_getsockopt() to get the error, is hanging inside getsockopt().

My IRX is able to connect to my host by using the lwip_* functions directly, so something funky might be going on with libsmb2. My code so far is over here. I'm gonna take another look over the next few days and I'd also be interested in collaborating.

Thanks @sahlberg for bringing the IOP stuff in libsmb2 into master. I wish I noticed your ps2 branch before I spent a decent chunk of time last week on the IOP gcc 3.9 compile/link errors! :)

[sahlberg]

Thanks @sahlberg for bringing the IOP stuff in libsmb2 into master. I wish I noticed your ps2 branch before I spent a decent chunk of time last week on the IOP gcc 3.9 compile/link errors! :)

No problem. I have an almost working smb2man.irx that I am working on. I should have it in a state so that opl can do its browsing of files after the weekend and can publish the sources then if you want to help out.

It works so far that in devctl I have a logging system where I connect to an smb2 share and open a file for logging. When you can do SMB_LOG("foo returned rc:%d\n", rc); from within the IRX then the debugging becomes so so so so much easier.

[sahlberg]

Email me and I can send you the sources for smb2man I have so far.

[sahlberg]

I have created SMB2MAN irx and it is avaliable at https://github.com/sahlberg/ps2sdk/tree/smb2man Maybe this would be useful to get into upstream ps2sdk.

It builds an IRX that is ~165kb in size, but this is the full blown libsmb2 so there is space to make this smaller by slimming down libsmbs. For example getting rid of the DCERPC layer as well as all the strings used in the nt-error to string mappings. (these strings take up a surprising amount of space)

Using a modified OPL, that links with this SMB2MAN instead of SMBMAN and just calls the CONNECT devctl (which is the equivalent to connect/logon/... in smbman) I can get OPL to show my game collection and the cover art. One thing, if someone wants to try it out with OPL, the way SMB2MAN works is that is creates one single smb: instance and underneath this instance you will have all the shares represented as one virtual subdirectory for each share. So if you have a share smb://10.10.10.11/MyShare that you named "NAS" in the CONNECT devctl, this share will show up as smb:/NAS/ If you add another share by calling CONNECT again and give it the name "SAN" then that share will show up as smb:/SAN/ This means that the root of the share is no longer smb:/ but smb:/NAS/ and you will have to go to settings in OPL and set Eth Prefix Path to "NAS" in order to access the NAS share.

You will need to do some hacking on OPL to even get it to use smb2man but it is not too hard to do. I can send a temporary hack/patch I used with hardcoded share settings if someone wants to try. But it is probably better to have someone that knows opl to do a proper patch instead.

My hacked up OPL can not launch any games. It just goes black screen and hangs. Maybe the irx is too big? Maybe there are other bugs? Have no idea. It should be possible to shrink libsmb2 down quite dramatically from the current 165kb IRX. Anyone wanting to hack on this, go ahead, I won't have anymore cycles to work on this in the forseeable future though.

[auanasgheps]

I can get OPL to show my game collection and the cover art.

This is an astonishing result. Thanks for keeping us updated!

[sahlberg]

So I will try to describe what I think needs doing now. I am not sure if this actually can/will work, it all depends on :

• can you open an lwip socket from one IRX, then unload that IRX without closing the socket. And this socket still remains open after you re-load a different IRX? Someone with IOP knowledge would need to step in here and say if this is possible or not. (i.e. basically, does the os running on the iop have resource-tracking for the irx modules it loads and unloads?)

IF that is possible. I think one way to get SMB2 into OPL is to: 1, Use the current SMB2MAN I posted above. This is a full blown libsmb2 IRX that is 165kb in size. If OPL loads this and uses it it for just the browsing of games then it shouldn't matter that it is this big. 2, When we want to launch an actual game, OPL will open the file. Then we will need a new simple DEVCTL to call into the IRX to fetch the pointer to the smb2 context as well as the pointer to the smb2fh structure for the file. these two pointers are what we need in order to later generate a smb2_read() operation. 3, Then from OPL we "simply" unload the SMB2MAN IRX without even shutting down the smb2 context. Leave it allocated and hanging in memory.

4, Load a new smaller IRX. This would be a tiny version of SMB2MAN and libsmb2 which only implements the lseek and read fucntions. My supersmall branch of libsmb2 can illustrate how you can cut down libsmb2 like that. For this I don;t think we need to use a special libsmb2 branch but instead just have a SMB2MAN-TINY IRX where we simply copy the chopped up libsmb2 sources into the irx src directory. No need to have a branch in upstream libsmb2 for this. 5, This new IRX needs a new DEVCTL call where we pass the two pointers from step 2 back to the module. 6, Then have the lseek() and read() methods always use these smb2 context and smb2fh pointers.

That I think has potential to work, but it depends on that a filedescriptor and allocated memory block will remain available even after the IRX that opened it is unloaded.

I won't have any time to work on this myself for the forseeable future so someone will have to test it. But first, anyone that knows how IOP programming work can tell if this can actually work? the crucial part is to be able to transfer an open filedescriptor across an IRX unload/re-load.

[Nerdy314159265]

I wanted to say @sahlberg it's great to see so much progress on this and reading this has been very interesting (not that I can fully grasp it).

I wanted to add on to what @auanasgheps said and mention also that SMB has some pretty major advantages for multi console environments since you can use one storage location to feed multiple consoles and removes unnecessary duplication of data/effort. Also it allows for more interesting setups, such allowing someone to move from real PS2 to emulated PS2 somewhat seamlessly (now that PCSX2 supports OPL VMC files). I personally find that the most thrilling prospect.

Speaking of VMC files, it sounds like the version of libsmb2 that would be running while a game is running will currently lack write functionality. I'm unsure of how the VMC system works (I'm trying to read and figure it out but I'm new) but I would think that would break VMC on SMB functionality since when the game would go to save it would be unable to write to the share hosting the file.

[dbmgr]

I just came to tell you that I admire your work and knowledge . I wish I could help you even a little but I don’t know anything about ps2 programming.all be safe and do what you really love!I m waiting patiently to see if any progress ll be made on this subject because it ll be really dope!

[joshuaseltzer]

Is SMBv1 still a requirement for the server hosting the Samba share? Has there been any more progress on this task?

[sahlberg]

I think SMB1 or SMB2/3 are probably the wrong solution for OPL. The protocol is stateful and does need a decent amount of code just to run. For libsmb2 I estimate that without authentication and without encryption, with decently big effort, you could probably get the compiled minimal object down to 60-80kb in size. Maybe. Which may be too much to spare for many games.

A much better solution and SMALLER client code would be the (is it still beta?/prototype?) UDPDB support. That will do all smb1/2/3 can do but faster and with significantly smaller footprint.