Add a max seed QA to SDG backend jobs

instructlab / instructlab-bot

GitHub bot to assist with the taxonomy contribution workflow

Apache License 2.0

14 stars 16 forks source link

Add a max seed QA to SDG backend jobs #239

Closed nerdalert closed 4 months ago

nerdalert commented 4 months ago

There is a hard timeout on jobs. Each job with an unknown load on the service is a regression of ~12sec/QA pair.
Hardcoding the max to 20 as a constant to keep avoid undue CLI bloat unless feedback says this will be a moving target.

Enhancement for #233

nerdalert commented 4 months ago

While testing this, I ran it against a knowledge PR with only 6 seeds. That job timed out. Granted it had more tokens than the skill PR I modeled time on but it makes me wonder if knowledge is more resource intensive or if it was just an outlier. Without visibility on the other side of the API, I am just guessing but it's worth asking around on.

nerdalert commented 4 months ago

GH is glitching out on the repo moves and closed this for me -_-