Open SolbiatiAlessandro opened 1 year ago
I agree, we should break the whole pipeline into smaller stages. I will try to add this soon.
@huyphan168 I made an attempt on imporving the pipeline infra here https://github.com/danielgross/LlamaAcademy/pull/10
Let me know what you think
Love this repo, would love to contribute!
Feels like the most needed feature is to have a good evals on couple of test docs (Notion, Github..) to evaluate the incremental models trained.
I also noticed it's non-trival to run the whole main.py flow as all the task are executed serially and if a task break you need to run the whole 1 hour script again. I would love helping to make the
docs_ingestion
,data_generation
etc executable in parallel so the infra can scale easier!Also open to any other suggestion for PR you guys have
curious what you guys think @huyphan168 @danielgross