federicoB
commented
2 years ago Hi! Very interesting post. You talk about a problem i noticed in the industry and i'm now trying to find solutions for. You post was helpful in confirming me that cookiecutter is the way to go but i can't not notice that you show real solutions only for problems 2 and 5 I give my 50 cents here:
- for the problem of git branches being overkill for simple parameter change you can you either do a manual method with a table file with all the parameters combination and results or use Sacred that is essentially the same thing but way more structured https://github.com/IDSIA/sacred
- Problem 1 can be solved with DVS
- Problem 6 solved by setting fixed seeds, etc...
- For testing, this is more complex, but should be made with metric threshold both in terms of predictions quality and performance. But those metric should be a function on the power of underlying systems.
Machine learning deserves its own flavor of Continuous Delivery | dlite.cc
An ML project ties together large datasets, a slow training process without clear pass/fail acceptance tests, and contains multiple types of deliverables. By comparison, a classical software project just contains code and has a single deliverable (an application). Thankfully, it’s possible to create a Machine Learning-specific flavor of Continuous Delivery (CD4ML) for non-enterprise organizations with existing tools today.
https://dlite.cc/2020/04/17/continuous-delivery-machine-learning-cd4ml.html