search

McSib / e621_downloader

E621 and E926 downloader made in the Rust programming langauge.
Apache License 2.0
60 stars 13 forks source link

Blacklist Update for 1.6.0 Release #34

Closed McSib closed 4 years ago

McSib commented 4 years ago

This is a published development and idea page on how the blacklist will be handled from now on. The blacklist in its current state is parsed 320 times when performing bulk searches, pool searches, and set searches. This issue will now discuss, and show the new change that will be done for the new update.

In light of the last issue created about the memory leak described here #33. The blacklist is clearly showing its limits now. Having been originally created for local client-side use, it didn't need any bells and whistles for how it should be implemented. But with the recent API change, the blacklist now needs to perform an API call for users, and because of this, it has broken its own rule on how it should be treated. The reflection of performance time has shown it clearly, waiting for long periods of time just to scan 320 posts, when it should take milliseconds. I have decided to redesign and take this to a new drawing board, giving the blacklist a better coat of paint.

With the recent light of having to develop this, I have made a design to reflect on how it will now function.

New Blacklist Diagram

Shown here is now the new set up for the blacklist. It will be stored in the WebConnector, and it will be the owner of the blacklist. The blacklist will be parsed and tokenized before being given to Grabber as a reference. This reference will be used to filter posts and bring far greater performance overall. Instead of parsing happening at runtime, it will now happen before grabbing even begins. This will give the program more freedom and breathing room to become more collected for the grabbing process. With hope, we will now have to perform another massive change for the blacklist again, unless the server updates and supports user blacklists server-side.

McSib commented 4 years ago

The blacklist system has now been added.