Setting the weights as constant allows us to avoid redundant pretranspose operations in Arm Compute Library (ACL) every time execute is called (they are now run once and cached). This delives big speedups especially for relatively small matmuls.
Note that this is a temp fix that needs to be handled carefully by primitive caches in frameworks, since the ACL object is now holding more state - i.e. we want to make sure that the cahce maps a layer with a specific set of weights to the oneDNN primitive storing those weights.
We're currently working on the proper fix for this which involves making lowp_gemm stateless and fixed-format in ACL and oneDNN.
Fixes # (github issue)
Checklist
General
[ ] Do all unit and benchdnn tests (make test and make test_benchdnn_*) pass locally for each commit?
[ ] Have you formatted the code using clang-format?
Performance improvements
[ ] Have you submitted performance data that demonstrates performance improvements?
New features
[ ] Have you published an RFC for the new feature?
[ ] Was the RFC approved?
[ ] Have you added relevant tests?
Bug fixes
[ ] Have you included information on how to reproduce the issue (either in a github issue or in this PR)?
Description
Setting the weights as constant allows us to avoid redundant pretranspose operations in Arm Compute Library (ACL) every time execute is called (they are now run once and cached). This delives big speedups especially for relatively small matmuls. Note that this is a temp fix that needs to be handled carefully by primitive caches in frameworks, since the ACL object is now holding more state - i.e. we want to make sure that the cahce maps a layer with a specific set of weights to the oneDNN primitive storing those weights. We're currently working on the proper fix for this which involves making lowp_gemm stateless and fixed-format in ACL and oneDNN.
Fixes # (github issue)
Checklist
General
make testandmake test_benchdnn_*) pass locally for each commit?Performance improvements
New features
Bug fixes
RFC PR