[FEAT]: Switch from jcifs-ng to smbj to increase SMB throughput

[Pentaphon]

Description

A long term solution would be to switch to https://github.com/hierynomus/smbj because the dev of jcifs-ng, our current smb backend is limited to 30MB/s which is the main reason why 4K and Bluray-bitrate video has been struggling to keep up for Nova users for some time. Now, admittedly, the new configurable buffer setting has been helpful by allowing more buffer to be stored, but eventually, as videos get higher quality (8K and so forth or higher bit-rate streaming becomes a thing to keep up with faster ISP speeds) Nova will begin to hit a stumbling block due to the current 30MB/s limit that will put it as a major disadvantage.

Now, the jcifs-ng dev has stated in the aforementioned issue that multi-credit requests can possibly double our current speed but even he cannot exactly predict what improvement it will make in real world use, and that any meaningful performance increases on par with Samba would require a full re-write of jcifs-ng and implied that such a re-write will never happen.

Therefore, in order to futureproof Nova and avoid any issues in the future, I suggest we make it a high priority to switch to smbj, or whatever implementation has the highest SMB performance on Android.

Additional information

N/A

[courville]

This is being considered. I will give a shot into integrating smbj after I am done with webdav support (usable now and supports huge 4k video throughputs). Note that smbj is SMB2+ only thus I will keep jcifs-ng as well not to cause regressions to SMB1 users (yes they exist and are numerous).

[GregBragg]

@courville I found an interesting wrapper to SMBJ here -> https://github.com/swaechter/smbjwrapper

It might be of interest to you in your investigations.

[courville]

@GregBragg interesting indeed. Thanks for bringing this to our attention.

[GregBragg]

@courville I'd be interested on how the port over goes to SMBJ... I considered taking this on for Spring Integration before v6.0 came out as it's a breaking change, however it's not entirely an easy endeavor.

If your game to improving jcifs-ng in regards to performance improvements, Moritz is more than willing to take on a PR that you author. I'm in the process of getting symlink support in this library now with his help, however his time is limited these days.

[courville]

Let's follow smbj integration in dev/smbj branches (stating slow).

[Pentaphon]

If your game to improving jcifs-ng in regards to performance improvements, Moritz is more than willing to take on a PR that you author. I'm in the process of getting symlink support in this library now with his help, however his time is limited these days.

It's probably best for the devs to cut their losses and go with something that is guaranteed to meet their needs. Especially if the jcfs-ng is limited on time.

[phhusson]

How is it guaranteed?

On April 8, 2023 7:09:51 AM GMT+02:00, Pentaphon @.***> wrote:

If your game to improving jcifs-ng in regards to performance improvements, Moritz is more than willing to take on a PR that you author. I'm in the process of getting symlink support in this library now with his help, however his time is limited these days.

It's probably best to cut our losses and go with something that is guaranteed to meet our needs. Especially if the jcfs-ng is limited on time.

-- Reply to this email directly or view it on GitHub: https://github.com/nova-video-player/aos-AVP/issues/835#issuecomment-1500793662 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

[Pentaphon]

How is it guaranteed?

It has much higher throughput than the current jcifs-ng implementation which Nova needs in the long run for ever increasing video bitrates.

[GregBragg]

How is it guaranteed?

It has much higher throughput than the current jcifs-ng implementation which Nova needs in the long run for ever increasing video bitrates.

Do we know this for certain? I heard the same when it was brought up in the Spring Integration project, however nobody has given me any proof of that yet.

Do you know where in the code it is slow?

[GregBragg]

If your game to improving jcifs-ng in regards to performance improvements, Moritz is more than willing to take on a PR that you author. I'm in the process of getting symlink support in this library now with his help, however his time is limited these days.

It's probably best for the devs to cut their losses and go with something that is guaranteed to meet their needs. Especially if the jcfs-ng is limited on time.

My PR is quite extensive, we have been working on it for a while now… if you have an idea where the slowness lies and you can contribute a fix that is not too extensive then it will probably be approved quickly.

[Pentaphon]

Do we know this for certain? I heard the same when it was brought up in the Spring Integration project, however nobody has given me any proof of that yet.

Do you know where in the code it is slow

People have been comparing them for years. I can't show you anything in the code. Just other posts on github and my own experience when using the current Nova builds on high-bitrate videos.

https://github.com/AgNO3/jcifs-ng/issues/104 https://github.com/AgNO3/jcifs-ng/issues/106 https://github.com/AgNO3/jcifs-ng/issues/263

[courville]

@GregBragg, mbechler highlighted that until multi-credit requests are implemented, jcifs-ng will causes issues playing heavy videos with nova. This is the task to be performed. Plan on my side is to keep jcifs-ng for SMB1 and use smbj for SMB2/3 until jcifs-ng has this enhancement. I noticed your work on symlinks, but on my side I have no plan to contribute to jcifs-ng beyond helping in reporting bugs helping to make jcifs-ng stable for the use cases I have (too much work for nova, too little time for other projects unfortunately). P.S.: and I can confirm the current throughput limitation of jcifs-ng compared to other protocols (sftp, webdav) that I measured. Let's see if smbj is any better.

[GregBragg]

Do we know this for certain? I heard the same when it was brought up in the Spring Integration project, however nobody has given me any proof of that yet. Do you know where in the code it is slow

People have been comparing them for years. I can't show you anything in the code. Just other posts on github and my own experience when using the current Nova builds on high-bitrate videos.

AgNO3/jcifs-ng#104 AgNO3/jcifs-ng#106 AgNO3/jcifs-ng#263

Yes I know, however if nobody is going to contribute a fix, don’t expect the maintainer to unless it’s in his best interests. Personally for SMB Spring Integration it has been sufficient for the almost four years I’ve been using it in Production for my use cases. Your use case is pushing the limits of jcifs-ng.

[GregBragg]

@GregBragg, mbechler highlighted that until multi-credit requests are implemented, jcifs-ng will causes issues playing heavy videos with nova. This is the task to be performed. Plan on my side is to keep jcifs-ng for SMB1 and use smbj for SMB2/3 until jcifs-ng has this enhancement. I noticed your work on symlinks, but on my side I have no plan to contribute to jcifs-ng beyond helping in reporting bugs helping to make jcifs-ng stable for the use cases I have (too much work for nova, too little time for other projects unfortunately). P.S.: and I can confirm the current throughput limitation of jcifs-ng compared to other protocols (sftp, webdav) that I measured. Let's see if smbj is any better.

Ok good luck… I found Smbj to be quite a pain to port it over from jcifs-ng, I think you are going to have a lot of effort to create a wrapper for it.

[Pentaphon]

Your use case is pushing the limits of jcifs-ng.

That's precisely why another implementation like smbj is sorely needed for a video player.

Ok good luck… I found Smbj to be quite a pain to port it over from jcifs-ng, I think you are going to have a lot of effort to create a wrapper for it.

If you can offer your help to Nova devs, we'd greatly appreciate it. Fast SMB throughput is a high priority for our network-reliant video player

[GregBragg]

Your use case is pushing the limits of jcifs-ng.

That's precisely why another implementation like smbj is sorely needed for a video player.

Ok good luck… I found Smbj to be quite a pain to port it over from jcifs-ng, I think you are going to have a lot of effort to create a wrapper for it.

If you can offer your help to Nova devs, we'd greatly appreciate it. Fast SMB throughput is a high priority for our network-reliant video player

I would but I’d be more amenable to getting jcifs-ng up to a point where it satisfies all use cases, which is why I initiated the PR for symlinks because it is beneficial to Spring Integration.

For the use cases I have currently, I write relatively small PDF files and read fixed length flat files which are performant with the current implementation.

I need to have a reason from my organization’s business point of view to expend the time on a feature that we don’t need.

[courville]

OK, I start having something usable (not releasable yet) and I confirm experimentally that there is no stuttering on heavy 4k video with smbj. I will keep jcifs-ng in nova for SMB1 and easy network shares discoverability. For now we can add smbj network shortcuts manually.

[Pentaphon]

OK, I start having something usable (not releasable yet) and I confirm experimentally that there is no stuttering on heavy 4k video with smbj. I will keep jcifs-ng in nova for SMB1 and easy network shares discoverability. For now we can add smbj network shortcuts manually.

Wow! You're making quick progress on this! From the way you talked about this I didn't think we'd see this happen until next year or something far off. Thanks for making this a high priority. Glad to hear this is much faster for 4K content. Looking forward to getting my hands on a release and start stress testing this on Ubuntu and Windows 11 SMB shares. This is definitely the most important step in Nova's development right now. I also suspect that supporting smbj will probably solve a lot of errors people have had with connecting to different newer devices.

Years down the road, you can probably drop SMB1 entirely when people get rid of their legacy devices and are forced to update to new devices and newer OSs that don't use SMB1 anymore. I personally find it surprising that people still use SMB1 but I guess a lot of people have been holding on to hardware longer than ever before.

[courville]

Note: cf. https://github.com/hierynomus/smbj/issues/586 smbj does not have name resolution implemented :facepalm: Consequence: for now users need to add manually network shortcut and input IP address of SMB server. Not very friendly and will create issues with indexing when SMB server IP changes if there is not a static DHCP address assigned to SMB server.

[Pentaphon]

Consequence: for now users need to add manually network shortcut and input IP address of SMB server. Not very friendly and will create issues with indexing when SMB server IP changes if there is not a static DHCP address assigned to SMB server.

That's fine. Easy to do with any normal router's IP reservation feature. Kodi users are used to doing this, so we can easily do it too. Can't wait to try it out.

Just gave that issue a thumbs up so the dev will see it needs attention. I suggest other people reading this do the same.

[phhusson]

lol, nice joke, thanks for the laugh

[Pentaphon]

lol, nice joke, thanks for the laugh

Hey it doesn't hurt to just click a button. Whatever I can do, I'll do it.

[courville]

OK, I can use jcifs-ng discovery for name resolution on smbj.

What we are trying to convey as a message, is that we are trying to get smbj usage as seamless as possible so that average Joe can simply use nova without convoluted manipulation trying to be ubiquitus. There is still work to get there.

[courville]

Experimental pre-release not for the faint of the heart: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.2

[ghost]

How do I know that it's using smbj? When I do the normal discovery it's the old code, but when I 'add indexed folder by URL' it's smbj?

[Pentaphon]

Experimental pre-release not for the faint of the heart:

Amazingly quick rollout! My observations so far:

With Windows 11 passwordless shares: connecting to smbj:// requires the local share's IP, the folder name is no longer optional but required or it doesn't seem to find it, and the name of the share as the username and the password left blank.

I notice the performance is much better, and I can see the configurable buffer (I have it set to 100mb) fill up a lot faster than before!

With my smaller Ubuntu shares there's no issues at all. Wonderful! This is how Nova should have been for so many years!

How do I know that it's using smbj? When I do the normal discovery it's the old code, but when I 'add indexed folder by URL' it's smbj?

You have to connect manually by pressing "browse a network share" and pull the dropdown to "smbj" and fill in the details.

Of course, you also have to remove your old smb indexed folder and start anew with the smbj index.

[ghost]

You have to connect manually by pressing "browse a network share" and pull the dropdown to "smbj" and fill in the details.

Of course, you also have to remove your old smb indexed folder and start anew with the smbj index.

Thanks.

Indexing went extremely quickly on my pixel 7, but browsing Movies or Series the app crashes immediately when I open any movie or TV episode details page.

When I'm quick enough to play a file it just loads indefinitely.

[Pentaphon]

browsing Movies or Series the app crashes immediately when I open any movie or TV episode details page.

Interesting. Everything seems to be fine on my 4K Stick and my Onn UHD 2021 box but those are streaming devices. Maybe the Pixel's newer version of Android might be an issue.

[ghost]

browsing Movies or Series the app crashes immediately when I open any movie or TV episode details page.

Interesting. Everything seems to be fine on my 4K Stick and my Onn UHD 2021 box but those are streaming devices. Maybe the Pixel's newer version of Android might be an issue.

Same issue in my Samsung Tab A7. Can later on check my android TV box.

Sidenote/question: NOVA always! asks for local file access even though all my files are on my SMB share. Is it really necessary to force local file access? The app outright quits if it does not get access and that seems a bit ridiculous to me. If I only grant it access to one folder it keeps pestering during start to grant more access. Is there any technical necessary for this annoyance?

Edit2: Same issue on my Homatics Android TV.

[courville]

Thanks all for the testing and feedback. I can see your crashes on the sentry crashnalytics backend:

com.hierynomus.mssmb2.SMBApiException: STATUS_OTHER (0xc0000008): Close failed for SMB2FileId{persistentHandle=05 40 00 00 00 00 00 00}
    at com.hierynomus.smbj.share.Share.receive(Share.java:380)
    at com.hierynomus.smbj.share.Share.sendReceive(Share.java:359)
    at com.hierynomus.smbj.share.Share.closeFileId(Share.java:171)
    at com.hierynomus.smbj.share.Open.close(Open.java:92)
    at com.archos.filecorelibrary.smbj.SmbjFileEditor.lambda$getInputStream$1(SmbjFileEditor.java:73)
    at com.archos.filecorelibrary.smbj.SmbjFileEditor.$r8$lambda$fI9hcNHDAPBry5wjpZx0RnVZWVE
    at com.archos.filecorelibrary.smbj.SmbjFileEditor$$ExternalSyntheticLambda1.run
    at com.archos.environment.ObservableInputStream.close(ObservableInputStream.java:42)
    at com.archos.filecorelibrary.StreamOverHttp$HttpSession.run(StreamOverHttp.java:214)
    at java.lang.Thread.run(Thread.java:923)

And this one too:

java.lang.NullPointerException: Attempt to invoke virtual method 'boolean com.hierynomus.smbj.share.Share.isConnected()' on a null object reference
    at com.archos.filecorelibrary.smbj.SmbjUtils.getSmbShare(SmbjUtils.java:133)
    at com.archos.filecorelibrary.smbj.SmbjListingEngine$SmbjListingThread.run(SmbjListingEngine.java:84)

And this one as well:

java.lang.IndexOutOfBoundsException: null
    at android.net.Uri$PathSegments.get(Uri.java:1026)
    at android.net.Uri$PathSegments.get(Uri.java:1011)
    at com.archos.filecorelibrary.FileUtils.getShareName(FileUtils.java:325)
    at com.archos.filecorelibrary.smbj.SmbjUtils.getSmbShare(SmbjUtils.java:120)
    at com.archos.filecorelibrary.smbj.SmbjListingEngine$SmbjListingThread.run(SmbjListingEngine.java:84)

I will investigate.

[courville]

New release addressing the crashes seen on sentry: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.3 Please test and report.

[ghost]

New release addressing the crashes seen on sentry: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.3 Please test and report.

Sadly, the issue persists on my end.

[courville]

@jaqarll indeed I see that it still crashes:

com.hierynomus.mssmb2.SMBApiException: STATUS_OTHER (0xc0000008): Close failed for SMB2FileId{persistentHandle=01 40 00 00 00 00 00 00}
    at com.hierynomus.smbj.share.Share.receive(Share.java:380)
    at com.hierynomus.smbj.share.Share.sendReceive(Share.java:359)
    at com.hierynomus.smbj.share.Share.closeFileId(Share.java:171)
    at com.hierynomus.smbj.share.Open.close(Open.java:92)
    at com.archos.filecorelibrary.smbj.SmbjFileEditor.lambda$getInputStream$1(SmbjFileEditor.java:73)
    at com.archos.filecorelibrary.smbj.SmbjFileEditor.$r8$lambda$fI9hcNHDAPBry5wjpZx0RnVZWVE
    at com.archos.filecorelibrary.smbj.SmbjFileEditor$$ExternalSyntheticLambda1.run
    at com.archos.environment.ObservableInputStream.close(ObservableInputStream.java:42)
    at com.archos.filecorelibrary.StreamOverHttp$HttpSession.run(StreamOverHttp.java:214)
    at java.lang.Thread.run(Thread.java:1012)

[courville]

New release treating hopefully the issue...: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.4

[ghost]

New release treating hopefully the issue...: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.4

It seems to be partly fixed, some files seem to work now, while others still show the issue.

Edit: it seems to be random, not specific to certain files. So the same file sometimes works and is playable, and sometimes crashes when opening the file overview.

[courville]

@jaqarll thx for all the testing and sorry for the iterative process. :facepalm: I forgot one case in the previous fix. New test, new release, kind of CI/CD process without the unitary tests: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.5

[Pentaphon]

It seems to be partly fixed, some files seem to work now, while others still show the issue.

Edit: it seems to be random, not specific to certain files. So the same file sometimes works and is playable, and sometimes crashes when opening the file overview.

On which devices? I only saw crashes on my Motorola phone when I didn't use enter the optional folder name. Once I did that, it stopped crashing for me. I didn't get any crashes on my Android TV devices. Installing 6.2.5 now.

[courville]

@Pentaphon you are too slow there is another release (/me hides)...

[Pentaphon]

@Pentaphon you are too slow there is another release (/me hides)...

I saw it as soon as I posted. Running 6.2.5 and currently giving it hell to see if it crashes on any menu! So far, it's pretty solid on my TV devices!

[ghost]

@jaqarll thx for all the testing and sorry for the iterative process. 🤦 I forgot one case in the previous fix. New test, new release, kind of CI/CD process without the unitary tests: https://github.com/nova-video-player/aos-AVP/releases/tag/v6.2.5

You are the man!

So far not a single crash.

[courville]

Thanks for the feedback. Just coded a nova option that will enable to use smbj instead of jcifs-ng (limiting SMB capability to SMB2+, failing with SMB1). This will provide a way to switch between jcifs-ng and smbj without re-indexing movie collection and adding manually links. It will thus preserve the nice jcifs-ng based SMB servers/shares discovery.

[ghost]

Thanks for the feedback. Just coded a nova option that will enable to use smbj instead of jcifs-ng (limiting SMB capability to SMB2+, failing with SMB1). This will provide a way to switch between jcifs-ng and smbj without re-indexing movie collection and adding manually links. It will thus preserve the nice jcifs-ng based SMB servers/shares discovery.

Do I understand it correctly that I can now do the search as before and just need to turn on the smbj+ option to use smbj automatically? In the shares the link to the smbj is still shown as smb and smbj.

Also, I now seem to be unable to input the smbj link manually unlike before. I'm not sure if I'm doing sth wrong or if it's connected to the newest version of nova. That's why I did not open a new issue here yet.

[Pentaphon]

Do I understand it correctly that I can now do the search as before and just need to turn on the smbj+ option to use smbj automatically? In the shares the link to the smbj is still shown as smb and smbj.

You have to manually create an smbj share using the browse shares function and then add to library. Remove the old smb share first or you will have duplicate indexes.

[ghost]

I am no longer able to do that. When I try to add it manually, it is just loading indefinitely.

Edit: ok, found my issue and now it's working again.

[courville]

@jaqarll, just tested on latest github pre-release and smbj links are working for me (you need to specify port 445 and not another one). Anyhow, by selecting using new SMB implementation in nova settings, you should be able to use old jcifs-ng discovery and rely on smbj to browse and play old smb:// links. Please make sure you are using at least v6.2.7 pre-release since it contains some important smbj fixes (multiple shares support that was broken).

[ghost]

@jaqarll, just tested on latest github pre-release and smbj links are working for me (you need to specify port 445 and not another one). Anyhow, by selecting using new SMB implementation in nova settings, you should be able to use old jcifs-ng discovery and rely on smbj to browse and play old smb:// links. Please make sure you are using at least v6.2.7 pre-release since it contains some important smbj fixes (multiple shares support that was broken).

Is there any way to see whether smb links are used with smbj?

[Pentaphon]

just tested on latest github pre-release and smbj links are working for me (you need to specify port 445 and not another one).

@courville some observations with 6.2.7 smbj, which are a huge improvement over smb, by the way:

1) When going to network shortcuts > browse a network share:

a) you don't need to specify port 445 at all. You can leave it blank and it just uses that by default b) however, you DO need to specify the Path which is marked "(optional)" or else the "Server" prompt will crash(?) to the Nova main screen if you leave it blank. Therefore, it is no longer optional unless this crash(?) is fixed.

2) I notice SMBJ has excellent performance and I've gone through hundreds of videos with it but I do notice that very, very rarely in "folder mode" the player will freeze at the last frame of the currently playing video and get "stuck" instead of going to the next video until I press the back button to select the next episode in a TV series manually. I can't reproduce it, but its happened at least twice with the hundreds of videos I've gone through it so far. (the kids love their older Nick shows so they help me stress test it) I'm not sure if this is a bug with "folder mode" or SMBJ but this didn't happen with folder mode using regular SMB. It's a very minor issue that has happened 2 times out of literally dozens or even a few hundred videos but it's worth reporting.

Is there any way to see whether smb links are used with smbj?

@jaqarll it should show up when you check the indexed folder link or browse the properties of a video file to see the address of where it is stored.

[ghost]

@jaqarll it should show up when you check the indexed folder link or browse the properties of a video file to see the address of where it is stored.

if that's the indicator the new option is definitely not working for me. Still showing smb everywhere.

[courville]

b) however, you DO need to specify the Path which is marked "(optional)" or else the "Server" prompt will crash(?) to the Nova main screen if you leave it blank. Therefore, it is no longer optional unless this crash(?) is fixed.

Thanks for the feedback: indeed this is a bug. Fix underway.

[courville]

if that's the indicator the new option is definitely not working for me. Still showing smb everywhere.

For clarification sake, ticking the option in nova will make smb:// links processed by newer smbj SMB library instead of jcifs-ng one. In other words it does not require to have smbj:// links in this case.