jeancochrane
commented
2 months ago I did a bunch of investigation into taichi in https://github.com/ccao-data/model-res-avm/pull/236. TL;DR:
- Taichi makes the code more complicated than numba and does not appear to be significantly faster, even with GPU support
- We could speed up the comps pipeline 2x by simply using a bigger instance type (c5.24xlarge)
- I am not going to update the pipeline to use a new instance type right now, for a couple of reasons:
- Performance improvement work is still ongoing, and we may end up improving the algorithm and changing the instance type shortly anyway
- We may not want to use the same instance type for comps and model training, since their resource requirements are different; we will sort out this design, including the specific instance types for our compute environments, when we work on https://github.com/ccao-data/actions/issues/16 later this year
The comps algorithm currently takes a long time to run. It takes 12 hours even with a reduced search space. However, the algorithm is highly parallelizable. It's possible that using some simple GPU programming could massively speed up the algorithm and make it tenable to run for every model. Specifically, I'm thinking that a quick spike using Python's taichi could be a worthwhile side adventure if someone has a spare day/needs a break from other work.