replace indexing step with nearest-neighbor search - Githubissues

direct-phonology / dphon

uncover old chinese textual parallels based on sound

MIT License

12 stars 1 forks source link

replace indexing step with nearest-neighbor search #131

Open thatbudakguy opened 3 years ago

thatbudakguy commented 3 years ago

this is useful for two reasons:

it allows seeding non-exact matches, which in turn allows capturing potentially interesting matches where most (or none!) of the match is exact
it helps us solve the multiple readings problem if we represent sound content as a vector so that there is not a 1-to-1 relationship between graphs and phonemes. this would let us compare using more abstract metrics like cosine distance, instead of text-specific metrics like edit distance.

we could start by looking into any libraries that do locality-sensitive hashing, like datasketch or the popular annoy. there's a great explanation of LSH here and a detailed one related to document comparison in 3.4.1 of Mining Massive Datasets.

thatbudakguy commented 4 months ago

if we use a vector database as suggested by #152, we could get this type of search built-in.