Assemble list of quirks with our full text search engine (esp. punctuation)

joewiz commented 7 years ago

As discovered by @mcdanielhn, a search for Banquo will not yield documents containing Banquo's or Banquo’s. (Lucene's Standard Analyzer doesn't treat an apostrophe (straight or curly) as a word marking boundary, following the Unicode word boundary specification rules.)
- https://history.state.gov/search?q=Banquo
- https://history.state.gov/search?q=Banquo%E2%80%99s
A search for f-16 (the airplane) is effectively identical to a search for f 16 - and will yield documents containing f or 16. Making this a phrase search "f-16" helps, but returns results without the hyphen like f 16. Same with the proximity search, "f 16"~0.
A search for an acronym with a slash returns a Lucene parsing error. For example, s/s (Office of the Secretariat Staff, Department of State) error:
Syntax error in Lucene query string: Cannot parse 's/s': Lexical error at line 1, column 4. Encountered: after : "/s"
- https://history.state.gov/search?q=s%2Fs
A workaround is to use phrase or proximity search, but again we would get undesired results, e.g., U.S.S.R. and U.S.S. are matches for "s/s".
- https://history.state.gov/search?q=%22s%2Fs%22

@HistoryAtState/editors: Please post any more examples you know of, and we'll work with @HistoryAtState/existsolutions to try out different Lucene analyzers and/or add advanced search form controls to see what combination can produce our expected results.

joewiz commented 7 years ago

See https://github.com/HistoryAtState/frus/commit/fc3f8b36d9cda22e7771fe0786b909548927a285 for an experiment with stopwords.

joewiz commented 7 years ago

For more on stopwords, see:

Lucene's default list of stopwords: https://stackoverflow.com/questions/17527741/what-is-the-default-list-of-stopwords-used-in-lucenes-stopfilter
Pros/cons of stopwords: https://www.elastic.co/guide/en/elasticsearch/guide/current/stopwords.html (linked from the stackoverflow page above)
Example of how our use of default stopwords leads to unexpected/unwanted search hits, see https://twitter.com/awmarrs/status/893213357251203077

joewiz commented 7 years ago

Regarding the banquo's example above, I did some experiments on this some months ago:

https://gist.github.com/joewiz/99ce7ea1082bc4ad14f013da9ba2def5

Related issues:

joewiz commented 6 years ago

Search scope, index configuration, and plumbing issues

administrative-timeline: needs to be added to $search:SECTIONS
frus-history/articles: need index on tei:div, needs to be added to $search:SECTIONS
frus-history/documents: needs index on tei:body, needs to be added to $search:SECTIONS
other-publications/serial-set: needs to be added to $search:SECTIONS
other-publications/vietnam-guide: needs to be added to $search:SECTIONS
wwdai: needs index on tei:div

Search omissions

carousel and frus-history/events: have never been included in search, could be useful
frus: volume titles not searched, only document titles
pocom: searched only on persName, not on position title, org name, or country
hsg-shell/pages: contains substantive text in some HTML
tags: not searched or leveraged

joewiz commented 5 years ago

From Michael McCoyer:

Sending you another search “quirk” I just encountered (I hope when you encouraged me to send these along that you really meant it!) –

So, our common terminology in FRUS headers for the National Security Advisor is “President’s Assistant for National Security Affairs.” When I searched that syntax as a phrase in Kristin’s Human Rights volume (https://history.state.gov/historicaldocuments/frus1977-80v02), however, I received no hits. The phrase does, however, appear in a number of headers in the volume – e.g., Docs 4 and 16.

Is it possible that the search queries don’t search on headers, or that they search them differently? I will leave this puzzle in your capable hands…

The exact URL for his search that returned zero hits is:

https://history.state.gov/search?q=%22president%27s+assistant+for+national+security+affairs%22&volume-id=frus1977-80v02

However, I found that if I changed the apostrophe in "president's" from straight (') to curly (’), the expected hits suddenly appeared:

https://history.state.gov/search?q=%22president%E2%80%99s+assistant+for+national+security+affairs%22&volume-id=frus1977-80v02&within=documents&sort-by=relevance

Thus, it appears that our search engine treats the curly quote as a literal character - like a letter in a word - rather than as punctuation that should be dropped. We need to get the search engine to treat curly quotes as straight quotes.

joewiz commented 5 years ago

From @joshbotts via the mailbox:

Wanted to flag this mailbox inquiry as an instance where a case-sensitive search capability would come in handy. Searching for "goa" returns several hundred hits, but most of them are for abbreviations for "Government of ..." rather than the geographical entity in South Asia.

--> User story: I want to search for "Goa" and exclude hits that are upper case ("GOA")

plutonik-a commented 4 years ago

This comment https://github.com/HistoryAtState/hsg-shell/issues/255#issuecomment-476926820 has been already issued here -> https://github.com/HistoryAtState/hsg-shell/issues/289

plutonik-a commented 4 years ago

This issue has been spliced into different existing and new issues (including backlinks to this one):

Therefore closing this parent issue.

marmoure commented 1 year ago

@joewiz for searching for the term s/s we can escape the forward slash with a backward slash s\/s and this will pass without the nasty lucene error

HistoryAtState / hsg-shell

Assemble list of quirks with our full text search engine (esp. punctuation) #255