Future Work

[GavinMendelGleason]

Next steps:

• We can optimize index - this should improve recall
• Improve embeddings - this should improve results - better prompt engineering, more complete data.
• Playing with threshold / model - we can alter the thresholds

Possible more robust approaches:

• Using a more sophisticated LLM with prompting and response
• Just use a human with the results from the service
• Build an integration with marc-records to high quality embedding to obtain plausible candidates in order
• Use these embeddings to obtain a RAG
• Alternative Models (new embedding models are coming out...)
• More robust matching approach - we use longer-range cost metrics which include neighbours and relations.

[GavinMendelGleason]

It's also worth suggesting combining with a larger identity graph model.

[GavinMendelGleason]

• Incremental indexing

[GavinMendelGleason]

• Use an open source embedding model