Need a post check option that checks the updated model

instructlab / instructlab-bot

GitHub bot to assist with the taxonomy contribution workflow

Apache License 2.0

14 stars 17 forks source link

Need a post check option that checks the updated model #229

Closed oindrillac closed 6 months ago

oindrillac commented 6 months ago

We need a bit postcheck command that points to a newly trained model endpoint hosted on BAM

BAM model hosted service needs to point to the newly saved model checkpoint
The Bot needs to interact with the model with a "postcheck" command (or another suitable name) to evaluate how the newly trained model is doing on the input
The traigers/authors will then accept or reject the submitted PR and the PR will be merged or closed

oindrillac commented 6 months ago

~For context, an implementation of this tooling will be needed to for a complete end to end run to finish. We are running an e2e from last week for which we have a new model checkpoint, we cannot merge the PRs until we triagers have a way to evaluate the performance of the new model on the skills/knowledge they have submitted~

Correction: Last week's e2e was a pipeline test, but this is a really nice to have for this current e2e

russellb commented 6 months ago

duplicate of #127 -- i'll copy your comments over there

oindrillac commented 6 months ago

Actually do not copy https://github.com/instruct-lab/instruct-lab-bot/issues/229#issuecomment-2059506242 - it was indicated that this is not needed for the run to finish as the previous run was just a test of the pipeline

russellb commented 6 months ago

@oindrillac ok i'll remove that part