search

danielgross / LlamaAcademy

A school for camelids
MIT License
1.21k stars 77 forks source link

Do we need Evals? #9

Open SolbiatiAlessandro opened 1 year ago

SolbiatiAlessandro commented 1 year ago

Love this repo, would love to contribute!

Feels like the most needed feature is to have a good evals on couple of test docs (Notion, Github..) to evaluate the incremental models trained.

I also noticed it's non-trival to run the whole main.py flow as all the task are executed serially and if a task break you need to run the whole 1 hour script again. I would love helping to make the docs_ingestion, data_generation etc executable in parallel so the infra can scale easier!

Also open to any other suggestion for PR you guys have

curious what you guys think @huyphan168 @danielgross

huyphan168 commented 1 year ago

I agree, we should break the whole pipeline into smaller stages. I will try to add this soon.

SolbiatiAlessandro commented 1 year ago

@huyphan168 I made an attempt on imporving the pipeline infra here https://github.com/danielgross/LlamaAcademy/pull/10

Let me know what you think