caikit / caikit-nlp

Apache License 2.0
12 stars 47 forks source link

Add embedding task #224

Closed markstur closed 11 months ago

markstur commented 1 year ago

Adds an embedding-retrieval task to get embedding vector for a sentence.

Data model is based on Gabe's feedback in https://github.com/caikit/caikit-nlp/pull/39

This embedding service will be extended in separate PRs. With multi-task support. This service can also support sentence-similarity and reranking.

markstur commented 1 year ago

Tag @gabe-l-hart and will tag Anjali if I can find her.

markstur commented 1 year ago

rebased to catch up to main (and so I can stack PRs with multi-task support)

markstur commented 1 year ago

@gkumbhat anything I need to do to unblock this one? Not sure if it helps if click the "resolve conversation" buttons. Up to you.

markstur commented 11 months ago

@gkumbhat Updated based on feedback. I don't want to use some of the saver() code until it is fixed to be safe and even revert partial changes, but we did get some unnecessary stuff out of save(). Thanks!