Deleting articles is extremely slow, if many articles in feed (was: "BULK-Delete" would be nice)

[klartext]

In newsbeuter 2.7, when I delete many articles by pressing "D"-key for a while, the deleting is very slow. It's an older/slower computer, but the problem is, I would guess, that each delete makes a single access to the sqlite-database. (connect,delete,disconnect)

It would be better, IMHO, if all delete-operations would be done just in mem, and when leaving a feed (or forcing an operation / syncing with database), that then all delete-operations will be done at once.

This would make the delete operations much faster.

[dredmorbius]

In the spirit of "the mutt of newsreaders", tagging then acting would be a great feature. Tag by pattern / expression, then delete / enqueue / whatever.

[klartext]

Yeah! :-)

[klartext]

My assumption, that there is database-access that makes the deletion slow was a wrong assumption.

I did profiling the nightmare of deleting about 1700 entries from a feed that had about 8500 entries. It took about 1 second per entry (estimated, not measured). So I need more time to delete the rest of the entries... because I stopped that torture for now.

For comparison: Importing all 8542 entries from newsbeuter-sqlite-database into MariaDB took about 40 seconds on the same old machine with my selfwritten tool.

Handling the display and running the controller (handling keystrokes, I guess; did not looked into the sources very intensively), took most time...

analyse_03.txt

Please see the attached profiling.

[Minoru]

Wow.

I don't see any way to really improve performance of do_wformat, but I'm investigating to see if I can at least call it less often.

[Minoru]

Tagging feature request was split out into #235.

[klartext]

@Minoru there is not any documentation on the function, but lets assume it formats a line, or all lines that are in the feed... ...if it formats all lines, not only those that are really visible, then it means doing work that is not necessary but eats up time.

I have 73 lines displayed, and 73 * 117 = 8542. (funny, that was the number of items in the feed yesterday ... btw. by chance division without rest :) ). If the algorithm is quadratic (just guessing), and we have 117 times more lines formatted than we needed, well, thats, why it's becoming slow...

So, I would guess, calling formatting functions only on the really displayed lines would help here. I would suppose, something like a windowing could help here. (Format only those lines, that are displayed, leave alone all other lines.)

[Minoru]

FYI: it formats a line, yes. It is indeed called on all lines, not just the visible ones, but that's only because we don't know which ones are visible—STFL handles that stuff for us. The algorithm that calls do_format is linear; do_format itself is amortized linear, worst-case quadratic.

[Minoru]

Please try the code from bugfix/110 branch.

[Minoru]

I guess we'll have to test this in production :)

[klartext]

Using a feed with 4052 items. All items are marked read.

Then deleting...

ORIG newsbeuter (newsbeuter 2.9):

pressing "D" for about 20 sec. This deleted 1236 items. This took roughly 6 min 07

Then undeleted the items with pressing "D" again, to have same conditions

FIXed newsbeuter ('bugfix/110' aka 0dfa9bb9b902688874643952fa551543adc75173):

pressing "D" for about 20 sec. deleted 1242 items This took roughly 2 min 29

So, this looks much better now. Think it can be seen as fixed. Speed could be better, but the speed-problem seems not be related with deletion anymore (more general maybe; that would be a different issue then, not opened so far).

Thanks for fixing!

BTW:
The fixed one also has different appearance in the feed-overview. Wheras the 2.9 shows all feeds normal, the 'bugfix/110' shows the feeds with unread items in bold. Looks like the new behaviour is more accurate.

Guten Rutsch! :)

[Minoru]

That's great news! I've opened separate issue so that I don't forget to look deeper into performance issues once bug count is down to zero.

Feeds with unread items are highlighted because listnormal_unread and listfocus_unread now apply to feedlist as well (see #222).

Щасливого Нового року! ☺