search

raquo / hnapp

Hacker News faceted search engine, RSS & JSON feeds
http://hnapp.com
MIT License
60 stars 10 forks source link

Unable to search for "extension type:story" #8

Open cklouman opened 2 years ago

cklouman commented 2 years ago

Thanks for a great service, only one problem so far: the above search term invariably results in "Server error" message. No other term that I've used causes this behavior. Grateful if you could check it out.

image
raquo commented 2 years ago

Hi, so there are a couple problems here:

"Server Error" happens when the search query times out. This tends to happen when the mentions of search terms are rare. It's been like this forever, and I don't know / remember enough about postgresql full text search indexing to say exactly why, or how to fix it. I suspect I'm doing something wrong there.

You can work around this by querying for "extension score>1" instead, which is (almost) equivalent because only stories (not comments) have scores. I think this works because postgres uses this higher-cardinality index prior to filtering the results by the full text index, or something like that.

Which brings us to the second problem – the query I suggested does not time out, but it returns no results. I don't know why that is. It seems to be specific to the "extension" search term. Searching for "chrome score>1" does return results have the "extension" word in them, so the data is there. "extension" does not appear to be a postgresql reserved word, and anyway searching for "table score>1" does return results, so even if it was, it shouldn't be an issue.

Hnapp's postgresql schema is written out in one of the project files, if anyone knows how to solve either of these issues, PR is welcome. The existence of this issue bugs me, but sadly I don't have the availability to figure it out.

--Nikita

cklouman commented 2 years ago

Ok thanks! I'm no expert on postgresql fulltext things either so can't help you there atm. One thing I noticed is that the timeout seems to occur for all search terms beginning with "ex", maybe that could be of help to someone with more insight. I'll just avoid those terms in my queries for the time being.

raquo commented 2 years ago

Seems to not be the case, I tried "external score>1" and "experiment score>1" and both returned results (one of them took a few seconds, but still). Which ex-word didn't work for you?

cklouman commented 2 years ago

OK right, I was not clear; I did not use the 'score>1' query in my tests, I tested using the 'type:story' syntax, and all such that starts with 'ex' fails for me. Which seems it could be useful for someone trying to debug the problem.

For all other terms (even those beginning with 'ex') except "extension" I can use the 'score>1' syntax and get the valid results. Sorry for the confusion.

raquo commented 2 years ago

Ah. It's not specific to "ex" words, but to words that are rarely used in the set of HN items that you're requesting - that first problem that I mentioned. For example both "ex" terms that I mentioned are also queryable with the "type:story" syntax, but for example "extreme type:story" times out trying to find the first 30 matching items, because there are very very few stories that match this, so it needs to look at ALL of the data in the DB, which fails because the query plan or indexing is probably screwed up in some way. If the index was properly configured I think it wouldn't need to do what feels like a table scan.

Message ID: @.***>

--Nikita