Try a database approach to ER

UHaifa-IS / whgazetteer-mehdie

World Historical Gazetteer - MEHDIE version

http://whgazetteer.org

BSD 3-Clause "New" or "Revised" License

1 stars 1 forks source link

Try a database approach to ER #193

Open tomersagi opened 4 months ago

tomersagi commented 4 months ago

Instead of batch matching, for a given dataset - embed all place names and then use a vector database to query the most similar embeddings. Then use a second model to predict the match using all the info.

tomersagi commented 4 months ago

[x] Find and install a vector DB
[x] Embed and fill all current places into the DB
[ ] Find a method to stack additional info (place type, location)
[ ] Train a simple matching model (SVM?) for different patterns of additional info
[ ] Write an experiment to test on existing testsets.
[ ] Write code to embed all new places upon ingestion
[ ] Write code to replace matching method in system.

tomersagi commented 4 months ago

[x] replace embedding model with mBERT

tomersagi commented 3 months ago

[x] extend the type similarity matrix

tomersagi commented 3 months ago

[ ] replace huggingface API with local model

https://cookbook.chromadb.dev/embeddings/bring-your-own-embeddings/

tomersagi commented 3 months ago

[ ] test different models on retrieval tasks - aim for 100% recall@k
[ ] run prep on the data before filling the db and add the variants and phonetics to the metadata.
[ ] add syntactic and phonetic features