Add fragma chat completion

Y-IAB / lm-evaluation-harness

A framework for few-shot evaluation of language models.

https://www.eleuther.ai

MIT License

0 stars 0 forks source link

Add fragma chat completion #22

Closed myeongho-jeong-yanolja closed 6 months ago

myeongho-jeong-yanolja commented 6 months ago

Updates

fix ill-defined header for fragma chat completion
Add retry loop for fragma chat completion and fragma translator
now we can use arguments that include "=" (ex. "endpoint=fragma.prod.in/openai/completion?model=gpt-35-turbo...")

hist0613 commented 6 months ago

The codes look redundant. Couldn't it be improved?

rifqiyan commented 6 months ago

FYI, axolotl and lm-eval-harness repos are migrated to Yanolja org, but it's marked as private repos instead of a fork. Do you guys still want to keep using Y-IAB repos instead of Yanolja ones?

The trade-off of switching to Yanolja:

It's harder to clone because it's not a public repo, so you need to setup credential if you want to clone it.
It's harder to sync with upstream, since it's not a fork.

What do you think?

myeongho-jeong-yanolja commented 6 months ago

FYI, axolotl and lm-eval-harness repos are migrated to Yanolja org, but it's marked as private repos instead of a fork. Do you guys still want to keep using Y-IAB repos instead of Yanolja ones?

The trade-off of switching to Yanolja:

It's harder to clone because it's not a public repo, so you need to setup credential if you want to clone it.

It's harder to sync with upstream, since it's not a fork.

What do you think?

I think sync with upstream is important, becuase original repository updates new tasks that may be usable for us.

myeongho-jeong-yanolja commented 6 months ago

The codes look redundant. Couldn't it be improved?

OK, I'll change retry loop!

rifqiyan commented 6 months ago

I think sync with upstream is important, becuase original repository updates new tasks that may be usable for us.

Okay, I will also keep working with this repo instead of the other one.