genesis-ai-dev / codex-editor

Codex Scripture Editor and Translator's Copilot
https://codex-editor.gitbook.io/
MIT License
7 stars 7 forks source link

Parallel passages extension #23

Closed ryderwishart closed 4 months ago

ryderwishart commented 7 months ago

We should leverage the copilot RAG index to reveal similar passages both in the translation draft and in the source text

Once we have this basic display working, let's try the following:

Image

dadukhankevin commented 7 months ago

A couple of things I'm thinking about.

ryderwishart commented 7 months ago

A couple of things I'm thinking about.

  • I'm not sure if embeddings will be accurate enough for order mismatch to matter. We'll have to see.
  • Does having this specific LAD on in the background all the time require too many resources? Switching databases, querying embeddings etc...

Yeah we need some kind of benchmark with 'accepted' translations basically.

As for resources, I suspect metrics other than semantic similarity will be quicker. I was thinking TF-IDF might be faster and perhaps even better for what we want. Eventually we want an ensemble of metrics that can be indexed in the background, ideally a lot of more traditional algorithmic ones.

ryderwishart commented 4 months ago

The most pressing need for this feature is to improve the UI. Currently there are overflow problems, spacing issues (results take up lots of space), etc.

See #54