alejandrogallo
commented
3 years ago Hi!
Thank you for this comment.
Quite frankly I'm a little bit puzzled by your timings, for me, doing
time papis list -a | wc -lgets me
962
real 0m0.411s
user 0m0.376s
sys 0m0.040sand doing a filtering
time papis list -a year:1981 | wc -lends in
6
real 0m0.538s
user 0m0.492s
sys 0m0.134sI do not wish to jump into hasty conclusions, but it kind of seems like you're not using the cache
time papis -s use-cache false list -a year:1981 | wc -lresults in
6
real 0m5.094s
user 0m31.306s
sys 0m0.290swhich would be coherent with your timings (kind of). I'd like to get to the bottom of this.
If needed be, I think a (rip)grep backend would be doable and useful. However I haven't come across the need for this.
Could you tell me what
papis config use-cachegives?
So that you understand exactly how the papis backend works, it's almost deceptively simple, in principle I create the list of all documents which gets created by reading every yaml file in every document. The parsing of the yaml documents is slow, but this is only done at indexing time. When this list is there I serialize it and store it in
~/.cache/papis/database/e700d592d0806fcab1fc70d2b6beac96-paperswhich is just a pickle serialization of the list. Later, reading
the documents is just done by deserializing this file, and it runs fast.
That's why I can't understand your timings, it would only make sense
if your use-cache is set to False, however I might be wrong
and it would be interesting if other people share their experience.
Cheers!
j-c-w
avonmoll
alexfikl
Hi all,
As my library size grows, I have noticed papis slowing down considerably. I have previously looked into making papis faster, but have come up short on ideas for how to fix the issue. Looking at performance profiles, the obvious way forward seems to be parallelization, but Alejandro has already tried that, and my own experiments with multiprocessing confirm what he found in (I can't find the issue number but I know I have seen it, anyway, Python multiprocessing doesn't seem to work to solve the problem).
The response times of papis are on the very slow end for me right now, I'm looking at 10+ second response times, which is getting to be too much. I have been using whoosh for some time, but it doesn't seem to skip the challenging part of this for papis, which is the file loading.
Now, I also want to point out that more performance should be possible. In my library of 737 items, running:
time papis list -a "year:2017"takes 11.99s, 11.80, 12.16Grep performs much better (on a much harder task as it also scans PDF files!):
time grep -nr '2017' ./*takes 5.47, 5.72, 6.05sExcluding PDF files from grep:
time grep -nr '2017' ./*.yamltakes 0.11s, 0.13s, 0.11srip-grep performs better yet:
time rg 20170.31s, 0.28s, 0.29So clearly, the disk speed is not the limiting factor. Perhaps someone with an understanding of Python file-handling can suggest some fixes? Or at least why it is slow and multiprocessing doesn't seem to help?
I'm happy to have a look at implementing ideas that seem sensible.
I do have some ideas to skirt this problem without addressing the real problem at hand (i.e. the slow file loading):
cat */*.yaml > all.yamltakes approx 0.2s.)I'm keen to hear other suggestions, and thoughts on the options I've suggested above, because this performance issue is starting to get a little bit out of hand in my library.
Jackson