Speed up Presto based on new Pytorch tools

WorldCereal / presto-worldcereal

4 stars 0 forks source link

Speed up Presto based on new Pytorch tools #30

Open kvantricht opened 8 months ago

kvantricht commented 3 months ago

@gabrieltseng I'm wondering what kind of speedup we could expect for this, mostly for inference. What's your perception of time it would take and the gain we get from it?

kvantricht commented 3 months ago

@gabrieltseng I did a local test with just an upgrade from torch==2.0.0 to torch==2.3.1 and had a 45% reduction in processing time. What would be the caveats to work with this version instead? From inference side, I think it's a no brainer to do the upgrade, results look identical.

gabrieltseng commented 3 months ago

Upgrading shouldn't be a problem, but we can add a test to make sure we are getting similar numbers

gabrieltseng commented 3 months ago

[x] Upgrade pytorch version done in #80
[ ] Investigate model compilation tools for speedup: https://pytorch.org/tutorials//intermediate/torch_compile_tutorial.html

kvantricht commented 1 month ago

This is being addressed in https://github.com/WorldCereal/presto-worldcereal/pull/88, however, before this can be safely merged, we need to:

[ ] add a test that confirms that original vs compiled encoder return the same result

Reason I add this is because I may have seen a case where predictions were different but this could have been due to some other thing. Therefore, it needs to be tested.