lookup: return all matching entries found during lookup

[cyphar]

Previously, we would just return the first entry we found that matched the requested word. This causes issues with dictionaries that have lots of entries which can be found using the same search string. In these cases, the user got a completely arbitrary word returned to them rather than the full set.

While this may seem strange, this is incredibly commonplace in Japanese and likely several other languages. In Japanese:

• When written using kanji, the same string of characters could refer to more than one word which may have a completely different meaning. Examples include 潜る (くぐる、もぐる) and 辛い (からい、つらい).

• When written in kana, the same string of characters can also refer to more than one word which is written using completely different kanji, and has a completely different meaning. Examples include きく (聞く、効く、菊) and たつ (立つ、建つ、絶つ).

In both cases, these are different words in every sense of the word, and have separate headwords for each in the dictionary. Thus in order to be completely useful for such dictionaries, sdcv needs to be able to return every matching word in the dictionary.

The solution is conceptually simple -- return a set containing the indices rather than just a single index. Since every list we search is sorted (to allow binary searching), once we find one match we can just walk backwards and forwards from the match point to find the entire block of matching terms and add them to the set in linear time. A std::set is used so that we don't return duplicate results needlessly.

This solution was in practice a bit more complicated because .otf cache files require a bit more fiddling, and also the ->lookup methods are used by some callers to find the next entry if no entry was found. But on the whole it's not too drastic of a change from the previous setup.

Fixes #30 Signed-off-by: Aleksa Sarai cyphar@cyphar.com

[cyphar]

@Dushistov This should be ready for review now, I've added integration tests as well.

[Dushistov]

The change is rather big, but I don't see any obvious flaws. So let's merge it into master, and wait bug reports. Thanks you for contribution.

[cyphar]

Thanks! Feel free to ping me if there are any issues. 😸