File list very slow on large torrents

[hypevhs]

Hi, I am running version 0.9.0a, the latest from PyPi. (If you would like me to retest using master, just ask)

I opened up a filelist of one torrent that had ~800 files in 20 directories. Just scrolling up and down takes 2.5s per each keypress. I typically expect this performance given that I'm running on a Chromebook, but I would imagine that it could have the same complexity as a terminal pager like less(1).

Is there any optimization that can be done to this view? Or any setting that would make scrolling and marking faster? Thanks.

[rndusr]

Yeah, that's expected. Sorry. stig isn't very snappy in general, but the file list is especially bad.

I tried to address any obvious bottlenecks, but the TUI file list code is quite complex, so I may have missed something.

It would likely be much more efficient if I removed the tree view and just displayed a flat list of file paths, but implementing that was quite hard and I really like it. There could be an option that enables/disables the tree view.

Filtering for specific files may help for now:

filelist 'the name of the torrent' 'files that contain this'

You can also filter files based on size or other criteria. See the "filter" help for a list.

[fn-ix]

Would it be any faster if the filelist opened with all folders collapsed, so that the user only expands the folders they need to? It would also make it easier to scroll through torrents with many files in several subdirectories, as right now the user has to scroll through every file to reach the bottom.

[rndusr]

It might be faster, but I'm pretty sure the issue is connected to urwid's ListBox. Either it's implemented inefficiently or I'm not smart enough to use it correctly. Large torrent lists (also ListBoxes) are also slow, and collapsing isn't possible for them.

I've decided to try prompt-toolkit to fix this and other issues.

[fn-ix]

Ok, good to hear some attention is being given to this! ;)

[Mitrandil]

any progress in that direction?

[rndusr]

Should be much improved thanks to @rsekman:

https://github.com/rndusr/stig/pull/249

Which version of stig are you running?

[rsekman]

Yes, #249 should have improved performance for large file lists. I'm noticing it in my personal usage with torrents with ≈ hundreds of files.

The change in #249 was to cache the widgets that make up the rows of the file list. This fixed the performance issue because it prevents the widgets from being re-created with each render call. However, the cache is a LRU (Least Recently Used) cache, so if the number of files exceeds the maximum size of the cache, the first rows will have been pushed out of the cache when the next render call comes around. The cacheing is then entirely ineffective, its tail is running away from us and we can never catch up.

In #249 I set the maximum cache size to 1024 because that's a nice round number and large enough for my usage. I tried lower numbers first and noticed that was not effective. I don't think we should let the che be unbounded because that could easily eat a very large amount of memory. Ideally it would be configurable. We'd have to do some intrusive runtime modification of the FileListWidget class but the ability to that is a feature of Python, not a bug, right?

tl;dr: are you running the latest version and are your file lists larger than the maximum cache size (1024)?

[Mitrandil]

@rndusr @rsekman okey, i understand, my version of stig is 0.12.10a0, today i check new version, thanks

[Mitrandil]

I check new version, yes it has much faster file listing. I hope your's project keep impoving. Thanks for support

[rsekman]

I’m happy the improvements I make for my own sake are also useful for others. That’s the spirit of FOSS, praise Stallman.