search

ruben2020 / codequery

A code-understanding, code-browsing or code-search tool. This is a tool to index, then query or search C, C++, Java, Python, Ruby, Go and Javascript source code. It builds upon the databases of cscope and ctags, and provides a nice GUI tool.
https://ruben2020.github.io/codequery/
Mozilla Public License 2.0
685 stars 86 forks source link

Slow to clear search result #52

Closed Masterivan closed 7 years ago

Masterivan commented 8 years ago

Hi,

I am working on a large code base. If I search in CodeQuery gui (windows 7) for a symbol and get like 20.000 results then the search is very fast around 1 second. But any search I do after getting such a large result takes around 1 minute. If I make the same search (that just took 1 minute) again then it only takes 1 sec (if the search give few hits). When having a result with many hits like the 20.000 I restart the gui (which takes a few sec), then any search is fast again.

My conclusion is that clearing the current result when making a new search is very slow somehow (if it contains many hits). Is that something you can do anything about?

Best regards, Ivan

ruben2020 commented 8 years ago

I'll take a look. Thanks.

Masterivan commented 8 years ago

Are you able to reproduce?

ruben2020 commented 8 years ago

Hi @Masterivan So far I couldn't reproduce it yet with a few thousand results. Need to find a bigger source base.

Masterivan commented 8 years ago

If I make a search that returns 2316 results it takes 3 sec first time, then if I perform the same search again it takes 10 sec. Search time probably also is affected by how big you you codequery.db is. In my case it is 1.1 GB. (my tags file is ~ 77 MB, and cscope.out is 228 MB.

ruben2020 commented 7 years ago

Hi @Masterivan I've tried with the Linux 3.10 kernel source code (excluding arch) and it generated a codequery.db file of size 1.7GB. I don't have performance issues. I suspect that it is related to sqlite. Now I'm using the latest sqlite. Try with the latest CodeQuery v0.19.0 on Windows. It's using sqlite 3.17.0.

ruben2020 commented 7 years ago

Let me test it further. Maybe it could be an issue.

ruben2020 commented 7 years ago

@Masterivan OK. I could reproduce it now and I know the root cause.

Masterivan commented 7 years ago

Hi Ruben, nice to hear that you can reproduce now and even found the root cause. What was the difference between when it was slow and fast?

Looking forward to see if you are able to do mitigate the issue.

ruben2020 commented 7 years ago

Hi @Masterivan The problem was with the clearing of the QTreeWidget. Turns out that I had been using it wrongly all these years. Although the QTreeWidget adds pointers to new'ed QTreeWidgetItems, there was no need to separately maintain and delete QTreeWidgetItems manually. It is deleted automatically by Qt when the QTreeWidget is cleared. It was the manual deleting of a large list (> 8K items) of these pointers that took a long time. With around 40K items, it would simply hang.

Now I just let Qt do it. With the commit to fix this, clearing a previous list of 835K items and replacing it with a new list of 835K items, takes only 3.2s on my PC. This excludes the time taken to query the database, which is about 4.6s on my PC. In this commit, I also changed some counting variables from unsigned int to (at least) long because unsigned int can only handle 65K while the Linux kernel 3.10 source base can produce millions of results depending on what you search for.

I think I can improve the performance further if I do the database query in one thread and the clearing of the previous list in another thread, but that would probably save only one second for 835K items which is not much of a gain.

ruben2020 commented 7 years ago

@Masterivan Please try it out and let me know if you're satisfied. Thanks.

Masterivan commented 7 years ago

Hi Ruben, Will there then be a 0.20 anytime soon to take advantage of your nice new improvements? Or how can I try it out before?

ruben2020 commented 7 years ago

Hi @Masterivan I will release it on Saturday. For now, you can try out a beta version here (download from here): https://bitbucket.org/ruben2020/codequery/downloads/cqsetup_001901_win32.exe Thank you. This site (bitbucket.org) seems slow and I did not verify checksums.

Masterivan commented 7 years ago

Hi Ruben,

I made the same 2 searches in the old 0.19 and the new 1901 version In the old (0.19) it took 99 sec to finish the second search. In the new 1901 it took about 1 sec, an I am very satisfied with the solution.

Br, Ivan

ruben2020 commented 7 years ago

Great. Thanks, Ivan! This will be released in 0.20.0 within 24 hours.