search

JohnSnowLabs / langtest

Deliver safe & effective language models
http://langtest.org/
Apache License 2.0
498 stars 39 forks source link

Preparing Embeddings Benchmarks (LangTest) #947

Open ArshaanNazir opened 10 months ago

JustHeroo commented 10 months ago

@ArshaanNazir please add your self as an assignee.

JustHeroo commented 8 months ago

@ArshaanNazir could you share your updates?

JustHeroo commented 7 months ago

@chakravarthik27 please share the latest updates.

chakravarthik27 commented 7 months ago

Hi @JustHeroo,

We have finished benchmarking the paul_graham dataset and now I am working on creating curated datasets for retrieval evaluation to do the benchmarking embedding models. I plan to generate question-answer pairs for each dataset and implement metrics to evaluate embedding models.

JustHeroo commented 7 months ago

@chakravarthik27 please update the latest status here.

Cabir40 commented 7 months ago

@ArshaanNazir could you share your update

ArshaanNazir commented 7 months ago

@Cabir40 Kalyan is working on it. He will be enhancing embedding benchmarks for other retrieval tasks. He is working on FinBERT-QA right now.

Cabir40 commented 7 months ago

@chakravarthik27 is there any update?

chakravarthik27 commented 7 months ago

Hi @Cabir40,

Still, it is in progress, I am currently working alone on the langtest project and implementing a high priority feature similar to Open LLM Leaderboard by Hugging Face(Eleuther). It may take some time to complete the embedding benchmarks.