[Feature Request] Better Android Support via anacrolix/torrent

[iamacarpet]

Hello,

I have been using Quasar for a while and it is great!

Last year I wrote a guide on XDA-Developers to get it working on the FireTV 2.

Unfortunately, the setup requires root as FAT32 is the only natively supported filesystem and Amazon have started shipping newer units with a not easily rootable firmware.

I've done a lot of research on how FAT32 support could be achieved and it started to look like the anacrolix/torrent library was the best bet, as it's 100% Go with modular storage drivers. They don't pre-allocate like libtorrent does, so fat32 isn't such an issue and with the modular drivers, it would be quite easy to write a driver to split the files into 2GB chunks to keep it under the 4GB hard limit.

Would the fact that it is Go and would simplify the build process, would Quasar consider it as a replacement to libtorrent?

Regards, iamacarpet

[KillerJoeBR]

I have no idea how much work this would take but I like the idea.

[KillerJoeBR]

But Quasar is already working great on Android as it is, almost no crash at all. The only minor issues for me right now are the watched status in library, and would be nice if episodes were added only after broadcast.

[iamacarpet]

Thanks @KillerJoeBR - I agree, it runs great, for me it's just the issue of needing to root if FAT32 isn't supported.

Did you have the same problem? And if so, did you find an easier way than the way I did it in my guide to mount the other file-system?

[KillerJoeBR]

Well i learned to live with the android limitations. I don't save any video and limit my searches to 4gb at most. And i use the internal storage. But i use a 32gb sd card where i storage kodi inside of it (so it doesn't fill up all my fire TV internal storage).

So i do see the advantages if i could use external drives for Quasar to download bigger files and i wouldn't have to worry too much about the internal storage free space. I could install more apps for example.

[iamacarpet]

I've done some research tonight and some testing on a FAT32 device.

I used go-peerflix with a modification to store the data in the current directory to make it easy to test on a FAT32 drive (USB thumb drive in this case).

For saying anacrolix/torrent is used in so many Android torrent clients, the FAT32 performance in the default file driver was quite poor, which was unexpected. Digging into it a little, while it doesn't use extents which prompts the OS to do a pre-allocation like libtorrent, it uses Go's ReadAt & WriteAt functions, which from what I can tell extend the file to as long as required to write the requested offset, which for torrents when a piece might be received that is 50MB ahead of our current handle and we don't want to trash it, still meant everything stopped while it wrote 0s.

Doing some further research led me to the BoltDB driver for anacrolix/torrent which stores each piece in it’s own DB entry based on the torrent info hash and the piece ID. While this doesn’t help us greatly as the BoltDB would also suffer from the same kinds of I/O problems, the key/value store driver approach looked promising.

This further led me to peterbourgon/diskv which is a disk backed key/value store, where each key (which in our case would be the torrent piece) is it’s own file, which would mean there would be a lot of files for a single torrent download, but the file-system limitations would be gone.

We would write pieces out of order without needing to worry about pre-allocation and the file-size limit wouldn’t really apply - the total file limit for FAT32 is 268,173,300 according to Wikipedia, which should be hard to hit.

Finally, I’ve seen at the moment you open your own file handle to the media file directly, which wouldn’t really be compatible with this method as the data would be jumbled. With anacrolix/torrent you obtain a seekable I/O handle via the client library itself, as demonstrated in go-peerflix.

This has the nice added benefit of dynamically switching the desired torrenting position to where we want to seek to, if we want to skip ahead, which would be a great feature! :).

[iamacarpet]

After writing that last night, I spent a little bit of time trying to make a demo.

I ended up writing a diskv driver for anacrolix/torrent, which out of the box didn't work great as I realised it wasn't writing whole pieces at a time, but wanting to write small offsets. This meant reading a whole piece, to change the required offset, then write it back in a true key/value fashion was causing a lot of I/O amplification.

Since we didn't really need the key/value style interface and it was just files under the hood anyway, I forked diskv and added a ReadAt and WriteAt function, which sped things up immensely. To be honest, now all diskv is doing is creating the folder structure and managing some locking around reading and writing, which could be brought directly into the driver in anacrolix/torrent.

Long story short, I ended up with 3 forks and a copy of go-peerflix (very small so easy to test with), which works with quite amazing performance on a FAT32 volume and gets around the 4GB file size limitations.

iamacarpet/go-peerflix

iamacarpet/torrent

iamacarpet/diskv

Test binaries are available at https://github.com/iamacarpet/go-peerflix/releases/tag/vdisk-test

There probably are a log of bugs as it's only a really quick proof of concept, I think the main one being that the pieces are all stored in a single folder so for huge files with a lot of pieces, this might cause problems - but diskv includes a transform function to organise them info subfolders if this'll help.

EDIT: Just did some additional testing myself and the extra bonus feature (the ability to skip ahead to say 40 mins in when you only have the first 30MB buffered) does work! It'll take a while to respond as it'll need to start downloading some of those further ahead pieces and re-buffer, which is currently very opaque, but after ~30 seconds it does just work.

[KillerJoeBR]

I have no idea about 80% of what you said but good work! Keep updating your progress! Thanks

[iamacarpet]

Cheers @KillerJoeBR

I'm a bit stuck at the moment, I've had a good search through the Quasar source but very complex and it's tightly integrated with libtorrent-go, as I suppose is to be expected.

The next stage is to switch libtorrent-go with anacrolix/torrent, but it looks like a change that will take a lot of time if you don't already understand the Quasar codebase, which I don't.

I don't currently have a lot of time to work on it, but I'll update again if I get the odd hour to make progress.

[KillerJoeBR]

Well maybe you should wait for scakemyer to come back and see what he thinks about it. Maybe he will help.

[elgatito]

Anyone is working on migration to anacrolix/torrent?

I'd make an optional memory storage, let's say about 200-500mb to store upcoming pieces downloads. It will fly even on poor boxes. Only could not seek quickly, but on the other hand - no disk needed if you don't want to store result.

[elgatito]

Discussion can be continued on https://gitter.im/QuasarHQ/Lobby

[elgatito]

Here are the binaries of Quasar, some few platforms that use gotorrent, instead of libtorrent.

What is NOT there:

• loading torrent files on startup (only start a new download, but if files were downloaded - it will check them).
• limiting speed.
• control from web.

Supposed test case:

• Find where is quasar binary in your Kodi homedir (for example, in linux-x64 it's - addons/plugin.video.quasar/resources/bin/linux_x64/quasar )

• Save existing binary to somewhat like quasar_backup

• Replace it with the one you have downloaded

• Restart Kodi

• Search for a movie/episode in Quasar, select a big enough stream and see how it works in your case.

android_arm.zip linux_x64.zip linux_arm.zip windows_x64.zip

[KillerJoeBR]

WOW i will try this big update later i will post my findings. Thanks

[elgatito]

I have posted new version here - https://github.com/elgatito/plugin.video.quasar/releases/tag/v0.9.78 Few bugs fixed and some functionality added, so you can give it a try and post your kodi.log in return.

[elgatito]

Version got updated. Link is the same to zip files. Should be almost fully functional. Can be tested as usual.

Some changes were imported from pending pull requests (they simplify current development).

[elgatito]

@KillerJoeBR, you can test watched state updates with this test version.

[iamacarpet]

I've updated the FAT32 storage driver, posted it in a new repo here (so it doesn't require a fork of anacrolix/torrent): iamacarpet/go-torrent-storage-fat32

There is also a new build and set of binaries for go-peerflix to test it out.

If @elgatito fixes the latest build of his gotorrent branch being stuck at "Checking" and not playing, I'll post some builds of Quasar with this storage driver enabled for testing.

[iamacarpet]

I've got a build of @elgatito 's gotorrent branch compiled for Android, but I'm having some issues installing it on the FireTV.

I keep getting errors on start and I think it is because the sdcards are mounted with "noexec", but that'd mean that no binary plugins would work on Android (locked down ones like the FireTV at-least). Haven't been able to pull the logs yet as I was doing the setup with a family member via a video chat, so I'll update with more info tomorrow, but the "noexec" thing looks likely.

Someone suggested you could re-package the main Kodi APK to include the plugin, but it would mean re-signing with my own key, so I'd need to re-package on each Kodi/Quasar update. I am willing to do that for the family for their particular use-case, but before I go to the trouble, @KillerJoeBR you said you had Quasar running on a FireTV ok using the internal storage?

I've posted the builds if anyone would like to try it themselves: https://github.com/iamacarpet/quasar/releases/tag/fat32-test

[iamacarpet]

Please ignore me, it wasn't anything to do with "noexec".

On inspection of the logs, it was a python error, but upon uninstalling and re-installing the same zip files from a different USB drive, it worked perfectly. Bad USB anybody?

Have personally tested that build on several FireTVs with storage on MicroSD using the FAT32 drive and it works preeeety sweet!

You might have trouble finding a writable folder, I think we used:

/storage/sdcard1/Android/data/com.xbmc.kodi/files/

[CancelIFR]

This is working fine for me, but I noticed the ability to use a proxy is gone. Any workarounds?

[foklorr]

😲😲😲😑😟 -=== NO ===-

fat32 is abominable. More than a third of the media I consume is in Matroska containers above the inane fat32 filesize limit.

Rally for UDF instead.

[iamacarpet]

@foklorr this works around the FAT32 size limit for devices that don't support UDF or other filesystems (such as Amazon's FireTV with microSD cards).

[talrhv]

Ok, so I download the android arm zip and replaced the files with the originals files on my Sony android tv. i exit kodi and reopen it. the result is quasar crushing loop. i've tried on on the latest version v0.9.78. did I missed something?

[deleder2k]

Does Quasar work on Fire TV "out of the box" now?

[iamacarpet]

@deleder2k check out Elementum. The change set became very large, so at @elgatito forked it.

[placidities]

it is long past time for android to support and use

UDF