TaehyeonKim-pyomu
commented
1 year ago This work proposes a method for determining the optimal value for the rank hyper-parameter in lightening the convolutional layer through low-rank tensor (matrix) decomposition using Bayesian optimization.
My proposed method can also be applied in Conv Layer lightening based on CPD (Canonical Polyadic Decomposition).
However, since CPD is inherently unstable in tensor decomposition, an additional optimization technique such as Alternating Least Squares (ALS) is usually employed for tensor decomposition.
The key contribution of this paper is to demonstrate that the optimal low-rank value for the decomposition of the convolutional layer can be found through Bayesian optimization. For this reason, and because Tucker-2 decomposition and SVD are more stable than CPD, these methods are used in the experiment section of this paper.
Furthermore, I would like to introduce two papers that have advanced this work:
[1] Kim, Taehyeon, and Yoonsik Choe. "Fast circulant tensor power method for high-order principal component analysis." IEEE Access 9 (2021): 62478-62492. [2] Kim, Taehyeon, Heungjun Choi, and Yoonsik Choe. "Automated Filter Pruning Based on High-Dimensional Bayesian Optimization." IEEE Access 10 (2022): 22547-22555.
Paper [1] proposes a new CPD-style tensor decomposition and includes experimental results using the Bayesian optimization-based Conv Layer Compression proposed in this work. Through this paper, you should be able to gain an understanding of the instability of CPD. Paper [2] proposes a high-dimensional BayesOpt technique for lightening all conv layers that make up a CNN, in an effort to overcome the shortcomings of the conv layer unit lightening technique.
Through these two papers, I hope to assist in-depth research activities in the fields of BayesOpt, tensor decomposition, and CNN compression.
I am pleased that you are interested in my work, and for further questions, please contact me at [taehyeon.kim@keti.re.kr] for a quicker response.
Why there is no experiments and implementation of CP decomposition rank selection? Paper itself mentions it, but Tucker-2 treats as more preferrable. Or do you have got any issues concerning CP optimization for CNN layers?