Extreme low performace on Android Tv-boxes with external disks

[lopezvg]

I’m extremely concerned with Torrest 0.0.15 performance on a high-end Android Arm Tv-box with external USB3 disks (both mechanical and SSD).

I just made a clean test to give you some figures (performed in two different boxes of the same model):

• TV-box: GTKing PRO, 6x Cores S922X, 4 GB main memory, 3xUSB3 120 MB/s read, 20 MB/s write, NTFS 5 TB disk, full Android 9 rooted, Kodi 19.5
• Comms: 500/500 Mbits optical fiber
• Time to access to torrent list within Torrest addon, while 3 torrents downloading: 4 minutes, where it was immediate in previous version
• Time to access to RESP server from Chrome to list torrents: similar to the access through Torrest addon
• Download throughput: about 20% of the previous version. Previously it reached the max disk write throughput (20 MB/s), where now the top is 4 MB/s
• I have tried with both the .so library and the binary with similar results.

With these results it is obvious that Torrest 0.0.15 is unusable in this environment and I’m force to switch back to the previous version while you identify the problem. I can do this task without problem, but I wonder what may be the feeling and reaction of other users in similar situation. Torrest has a great reputation among users, but can be drained rapidly if you do no address the issue rapidly.

NOTE: Just to complicate the picture, I have tried on my mobile, Motorola One Action (medium-low performance), Android 11, using the internal memory for download, with NO visible performance degradation, reaching 30 MB/s downloads speed (max mobile Wifi speed), with immediate response on listing the active torrents.

[i96751414]

Hello. Thank you for pointing that.

I have already noticed something similar to what you are describing and I think this is caused by libtorrent (https://github.com/arvidn/libtorrent/issues/7376). One of the changes from last torrest binary was the bump from libtorrent v1.2.16 to v2.0.8.

I'm currently very busy and can't spend much time with this, however I will try to have a look on this during the weekend. I will also try to downgrade libtorrent version to see if that's something that could at least temporarily solve this.

[i96751414]

Hi, could you please test this library and see whether it has the same issues? libtorrest.zip

It is an android-arm64 lib with libtorrent v1.2.18 - I didn't experience any problems on my side

[lopezvg]

I'm going to try it on my mobile with Android 11 and download path in the internal memory, but remember that in that case it was working fine. Could you please compile the android-arm7 lib so I can test it in a problematic environment?

[i96751414]

Sure, give me a few minutes

[i96751414]

Here it is: android-arm.zip

[lopezvg]

The test on my mobile with Android 11 and download path in the internal memory is OK, full speed, as it was with the official 0.0.15 version.

On the TV-box GTKing PRO with Android 9, much better. Performance on background downloads are back to normal. However on foreground reproduction, on Buffering, it is happening a weird thing: it starts buffering rapidly but on reaching about 97% goes back to 60 or 70%, once and again and again, until finally starts the reproduction or cancels (probably due to timeout on buffering).

So, much better, but the Buffering requires a look into.

[i96751414]

That means you are having hash failed (from libtorrent), making the progress to go back as the pieces are discarded. I will compile with an older version of libtorrent and send you the library to see if it persists

[i96751414]

Can you try this one? It's built with lt 1.2.16 android-arm-1.2.16.zip

[lopezvg]

I have done a quick test with similar results on the buffering issue. I use very seldom the foreground download, so probably the problem is there for a while...

[i96751414]

I'm not convinced the problem is there. You can try setting the alerts log level to trace, play a torrent, close kodi and then see what's there (from the userdata torrest.log). You will very likely see hash failed alerts

[i96751414]

I compiled torrest for an older version of lt. Can you try it please? android-arm-1.2.13.zip

[lopezvg]

Working on it... Give me a few hours to prepare a case-study...

[lopezvg]

I have done several tests in three different environments, attaching test files:

1. TV-box(1): GTKing PRO, 6x Cores S922X, 4 GB main memory, 3xUSB3 120 MB/s read, 20 MB/s write, NTFS 5 TB FRAGMENTED Seagate mechanical disk, full Android 9 rooted, Kodi 19.5
2. TV-box(2-a): GTKing PRO, 6x Cores S922X, 4 GB main memory, 3xUSB3 120 MB/s read, 20 MB/s write, NTFS 1 TB Defragmented SSD Samsung disk, full Android 9 rooted, Kodi 19.5
3. TV-box(2-b): GTKing PRO, 6x Cores S922X, 4 GB main memory, 3xUSB3 120 MB/s read, 20 MB/s write, NTFS 5 TB Defragmented WD mechanical disk, full Android 9 rooted, Kodi 19.5

All three disks are of high quality bought at Amazon Spain or similar. Verified the S.M.A.R.T. statuses and are all-green Defragmentation performed in Windows 10 with O&O Defrag Professional Comms: 500/500 Mbits optical fiber

1. TV-box(1) results:

   • torrest_0.0.15-standard-2.0.8: o Buffering: unusable o Download: 20% of rate o Listing Torrest torrents: 4 minutes response time
   • torrest_0.0.15+android-arm-1.2.18: o Buffering: acceptable, a lot back and forth o Download: good, normal o Listing Torrest torrents: immediate
   • torrest_0.0.15+android-arm-1.2.16: o Same as on torrest_0.0.15+android-arm-1.2.18
   • torrest_0.0.15+android-arm-1.2.13: o Same as on torrest_0.0.15+android-arm-1.2.18
   • torrest_0.0.13: o Same as on torrest_0.0.15+android-arm-1.2.18
   • torrest_0.0.12: o Same as on torrest_0.0.15+android-arm-1.2.18

2. TV-box(2-a) results:

   • torrest_0.0.15-standard-2.0.8: o Buffering: unusable o Download: much slower than normal
   • torrest_0.0.15+android-arm-1.2.18: o Buffering: good, with a bit of back and forth o Download: good, normal o Listing Torrest torrents: immediate

3. TV-box(2-b) results:

   • torrest_0.0.15+android-arm-1.2.18: o Buffering: good, smooth, NO back and forth o Download: good, normal o Listing Torrest torrents: immediate

My conclusion is that Libtorrent is way too exigent of disk performance. Libtorrent may operate badly simply because of disk fragmentation. I know how to defrag disks, but most of the people don’t. May be that are parameters that can be tuned to alleviate these situations.

In summary, you proposal to release an immediate new version with Libtorrent 1.2.18 seems the appropriate one, however, further investigation on the disk performance issue would be required.

For the new version, Android Kodi 18 users would have to be taken in consideration, dynamically selecting at run time to use binaries, leaving open the door to use library.so in case the user migrate to Kodi 19+.

Torrest - TVBox(1) with mechanical 5TB fragmented disk.zip Torrest - TVBox(2-b) with mechanical 5TB fully DEfragmented disk.zip Torrest - TVBox(2-a) with SSD 1TB fully DEfragmented disk.zip

[i96751414]

Thanks for the tests. Just one curiosity, are your disks formatted as NTFS?

I remember we had similar issues for quasar: https://github.com/scakemyer/plugin.video.quasar/issues/862#issuecomment-351429195

It may sound silly, but I dont have problems at all when I use kodi's path structure. Using a different path makes the libtorrent fail its hashes. I tested this on 2.0.8

[lopezvg]

Yes, all of them are NTFS formatted. Fat32 does not support large files and exFAT is not officially supported in Android. I tested it and it lasted a couple of hours before data corruption. I use NTFS since the Quasar times, years ago.

NTFS let me interoperate with Windows if needed, and provides the strongest security on data structure. Unfortunately it has a toll: the write performance is some 20% of the disk nominal rate. With exFAT I get 120 MB read / 100 MB write with the SSD disks, but formatting with NTFS only get 100 MB read / 20 MB write.

The Quasar issues you are mentioning are USB Pen-drive and Flash memory units related. I'm using full mechanical o SSD disks. Android manages USB Pen-drive and Flash memory units very badly.

What do you mean by "kodi's path structure"? Using "special//..."? If that is the case those special paths got to be translated to real addresses before use. The only point is that they always refer to internal memory space, which is normally Ext4 formatted and high performance.

[i96751414]

I understand what you're saying, but in my particular case I don't have any external storage device, however I have the same problem depending on the path I use for storing files. This suggests there is a strange behavior with the way paths are handled by libtorrent.

For instance, the following path works just fine: storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/userdata/addon_data/plugin.video.torrest/downloads

However storage/emulated/0/Torrest/Downloads has a ton of hash failed alerts.

[lopezvg]

That may have a meaning if you are using Andorid 10+, where Android system is very strict with paths owner & permissions.

Instead of using storage/emulated/0/Torrest/Downloads, try to use a standard-demilitarized folder, like storage/emulated/0/Download, and inside build you folder structure.

That is the case with my mobile phone, and I have used that approach and it works fine, even with the 0.0.15 standard version

[i96751414]

0.0.16 was just released. I believe we can now close this one.

[lopezvg]

Hi,

Installed the new version and while it is usable, still far from optimal To be honest I am very disappointed with Torrest behavior, remembering the good old times with Torrest developed in GoLang.

Play function in devices not high performance trends not to work. To start, the buffering goes back and forth many times, even in fully defragmented disks. Finally when it starts the play it stalls or frizzes the image as the sound continues. If the play is paused, the download speed shows good rate, but on replaying sometimes does not recover..

Honestly I do not have a clue of what is going on with Torrest, but definitely it has lost its freshness. I do not know if the problem comes form Libtorrent or it is the way you interact with it, but the result is far from optimal.

All this said is in constructive mood, because we need that Torrest goes back to the freshness it had.

[i96751414]

I think this is not related with the implementation being C++ or golang. Torrest uses libtorrent api to play stuff, not much has changed around that. On the other hand, many things changed from golang to C++ impl:

• the lib parsing http requests is not the same;
• the libtorrent version is not the same;
• the way pieces are read on libtorrent is not the same as well...

Anyway, I don't experience any issues (unless the storage related issue for android devices) and as such it is very hard to debug something I can't reproduce.

Also, it would be useful setting all the logs to trace and send a log when you experience these issues (player freezing, etc).

[i96751414]

An additional note: you could compile torrest with -Dlegacy_read_piece=ON as that was a big change from 0.0.3 version.

make -f docker/Makefile torrest-android-arm IMAGE_TAG=lt1.2-latest EXTRA_FLAGS='-Dbuild_library=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE="$${CMAKE_TOOLCHAIN_FILE}" -DBoost_USE_STATIC_LIBS=ON -DOPENSSL_USE_STATIC_LIBS=ON -DCMAKE_SHARED_LINKER_FLAGS="-static-libstdc++" -Dlegacy_read_piece=ON'

You can see more on https://github.com/i96751414/torrest-cpp/blob/master/docs/building.md for compiling torrest.

[lopezvg]

I would certainly try, but that does not resolve the major issue for thousands of user with similar situation that mine. I personally do not have a problem because I normally download everything in background for latter play

[i96751414]

What I meant is: if legacy read piece is faster/more performant I don't mind using it as default for torrest while libtorrent does not implement a proper cache around pieces. But as I said, I am not experiencing any issues so I need someone to tests this.

[lopezvg]

I must be doing something wrong. I did the following:

• Downloaded an extracted the latest Torres_CPP master
• run ./scripts/install_dependencies.sh from inside Torres_CPP master folder
• run, as you said, make -f docker/Makefile torrest-android-arm IMAGE_TAG=lt1.2-latest EXTRA_FLAGS='-Dbuild_library=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE="$${CMAKE_TOOLCHAIN_FILE}" -DBoost_USE_STATIC_LIBS=ON -DOPENSSL_USE_STATIC_LIBS=ON -DCMAKE_SHARED_LINKER_FLAGS="-static-libstdc++" -Dlegacy_read_piece=ON'

The result; the same!

See the attaches Torrest.log torrest.log

[i96751414]

The log has lot of hash_failed alerts explaining the issues with download, etc. This is completely handled on libtorrent side.

Also, I believe you didn't have the service logging level set to debug. That would be useful to confirm libtorrent v1.2.x is being used

[lopezvg]

I hope I did it right this time. Still I do not see Libtorrent version

torrest.log settings.zip

[i96751414]

Oh, I see! You have the user-agent defined as "0". Can you go to torrest settings and delete the user-agent string? If the string is empty, torrest will generate an user agent based on both torrest and libtorrent version.

[lopezvg]

What a difference! Curiously I have the same setting y several devices with differente results... Anyway tell me how we procede. Do we retest Torrest version 0.015 with user-agent resolved? How do we patch all users with similar setting? settings.zip torrest.log

[lopezvg]

A quick test with standar 0.0.16 and the buffering was perfect, however, on starting the play it frizzes with black screen. Pausing the play I could see the download rate was 20 MB/s torrest.log

[i96751414]

In regard to the user agent, the default is an empty string, however I think that probably due to the fact that it was an integer on previous versions (defaulting to 0) kodi didn't reset this setting. This could be fixed though with some custom code.

Regarding the player freezing, I wonder if libtorrent is using all the disk. Can you try limiting the download speed?

[lopezvg]

I limited the download speed with good results. Now it does not frizze. However still there are a lot of "hash_failed" on buffering with both the standard 0.0.16 and with the "legacy"pieces version (and agent=''). I did verify the SSD disk, defragmented it and nothing changed. I do not know what to say. I do not find any difference between the two versions. Checked with version 0.0.15 and still totally unusable. Torrest.zip

[i96751414]

Ok, that's progress. I'll compile with some older versions of libtorrent and provide you with the binaries so we see if there are any improvements

[i96751414]

One point we also need to keep in mind: the android NDK/API has also changed in these binaries and could introduce changes in their behavior. But this is something we can't rollback as new Android devices require an higher API version.