search

szarroug3 / X-Ray_Calibre_Plugin

X-Ray Creator plugin for Calibre
http://www.mobileread.com/forums/showthread.php?t=273189
GNU General Public License v3.0
57 stars 12 forks source link

Add caching for network operations #77

Open stoduk opened 7 years ago

stoduk commented 7 years ago

Much of the work done is talking to remote websites, and there are two issues here:

1) when testing it is really slow to always go and fetch the same set of files for each run. Having them locally cached would speed things up.

This could be done very simplistically - just save the files we open, never timeout caches, etc. We could have a flag to turn it on/off, so most users won't touch it.

2) slightly better would be to try to cache all operations globally - imagine if we are getting info for a large number of books in a series. Currently we are getting the authors and all characters once for each book, when this information won't have changed. We could cache that and speed things up plenty.

Would need to consider how to handle cache ageing/clearing, as well as where to store the cache.

Simplest would be to cache files across a single run, but that doesn't help issue 1 above so isn't great. Better would be to have a global cache directory (does Calibre have a directory like this we can use?) and we can store everythign there. Can give the user several options:

My gut feeling is we should find somewhere to have a global cache (to benefit both devs and users), nd only have the options to enable/disable caching, and a button to clear the cache. If the user finds information is stale (eg. because they've edited it on GR) then they can clear the cache.

Thoughts?

szarroug3 commented 7 years ago

Sounds like a good idea. I've considered this before but I wasn't sure how to do it. Correct me if I'm wrong but I'm assuming global caching would be so that if you cache a website, when I try to create files for the same book, I can go use the cache. If I'm understanding correctly, when using global caching, what does the clear cache actually clear? Maybe we can give them a way to pick a specific book to clear the cache for or something. I wouldn't want to clear out the cache for every single book just because one book got updated.

stoduk commented 7 years ago

For GR, the files we are talking about are pages for: books, authors, characters, book meta-data (tooltips and buttons/glide/)

If we store the data per-book (ie. in the 'Calibre Library/{author}/{book}/' directory) then it only helps people who run the script repeatedly for a given book, whiile not expecting the data to change. So largely you and me!

Lots of those pages we've got for one book will probably be common to other books - books by the same author, books in the same series (so same characters), etc.

So, if we store the data globally (not sure where - would top level in Calibre Library be allowed? Maybe some user specified directory such as ~anthony/calibre_data/?) then we get a lot more benefit.

It would be a faff to know what files to clear for a given book (eg. figuring out which characters/authors to delete), but it would be possible to have a per-book config option to say "ignore cache" so that we download fresh copies of the data if we know it might have changed.

I think doing it globally is probably the right answer - if we are going to the effort of caching we might as well get the most benefit. That just leaves figuring out:

szarroug3 commented 7 years ago

Ah, I get it now. Okay, we can probably go to the top of the calibre library which is where all the folders with all the books are stored. We're already getting that folder (because we need the path for the local file storage anyway). The variable is self.LIBRARY in book.py. I think we can create a folder in there called cache or something and store them in there.

As for the user control, I'm not really sure. I think that's something I'd probably figure out while I'm doing it. I think the toggle and one-time purge are both good. Where would you put the per-book config option? I'm guessing in the book specific preferences page?

On Tue, Nov 1, 2016 at 5:29 PM Anthony Toole notifications@github.com wrote:

For GR, the files we are talking about are pages for: books, authors, characters, book meta-data (tooltips and buttons/glide/)

If we store the data per-book (ie. in the 'Calibre Library/{author}/{book}/' directory) then it only helps people who run the script repeatedly for a given book, whiile not expecting the data to change. So largely you and me!

Lots of those pages we've got for one book will probably be common to other books - books by the same author, books in the same series (so same characters), etc.

So, if we store the data globally (not sure where - would top level in Calibre Library be allowed? Maybe some user specified directory such as ~anthony/calibre_data/?) then we get a lot more benefit.

It would be a faff to know what files to clear for a given book (eg. figuring out which characters/authors to delete), but it would be possible to have a per-book config option to say "ignore cache" so that we download fresh copies of the data if we know it might have changed.

I think doing it globally is probably the right answer - if we are going to the effort of caching we might as well get the most benefit. That just leaves figuring out:

  • where to put it, as mentioned above. Any thoughts?
  • how we let the user control it. I think having an enable/disable config toggle, plus a button to do a one time cache purge, plus then a per-book config option to allow them to specify to ignore the cache (probably rather than a config flag it is just a way to mark a book as 'dirty' so on the next run we'll download the files fresh and clear the 'dirty' flag)

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/szarroug3/X-Ray_Calibre_Plugin/issues/77#issuecomment-257718401, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEjUVlTwt01SRkMIICFS43cG4Iq-7o1ks5q5704gaJpZM4KmcDb .

szarroug3 commented 7 years ago

@stoduk I've finished up the pylint fixes; let me know if you see anything you think I need to change; i'm going to try to test it with a lot of books to make sure i didn't miss any bugs or anything.. i think you can start working on this issue though since there should be no more huge changes.. try to stay pylinted with your changes though!