search

Model-Compression / Lossless_Compression

We propose a lossless compression algorithm based on the NTK matrix for DNN. The compressed network yields asymptotically the same NTK as the original (dense and unquantized) network, with its weights and activations taking values only in {0, 1, -1} up to scaling.
https://proceedings.neurips.cc/paper_files/paper/2022/file/185087ea328b4f03ea8fd0c8aa96f747-Paper-Conference.pdf
Apache License 2.0
23 stars 2 forks source link

How to replicate #2

Open 0seba opened 1 year ago

0seba commented 1 year ago

Hey, I got an interest in your work and I'm trying to replicate the results. What I understood is that I had to run the performance_match.py file in the mnist folder, and setting the initialization_way to ternary and kesi to 0 I would replicate the ternary weights experiment. After doing this I got a validation accuracy of 43.5%, I also tried with kesi 0.5 and got a similar 44.5%. How should I properly go to reproduce the results?

Finally, this method seems useful to compress random feedforward networks, do you have any ideas or other resources on how to compress trained networks?

Thanks!

gly-no commented 11 months ago

Hey, I got an interest in your work and I'm trying to replicate the results. What I understood is that I had to run the performance_match.py file in the mnist folder, and setting the initialization_way to ternary and kesi to 0 I would replicate the ternary weights experiment. After doing this I got a validation accuracy of 43.5%, I also tried with kesi 0.5 and got a similar 44.5%. How should I properly go to reproduce the results?

Finally, this method seems useful to compress random feedforward networks, do you have any ideas or other resources on how to compress trained networks?

Thanks!

Hi! Sorry for the late reply as I didn't check the comments promptly. I checked our codes and found weight_num_list in the performance_match.py is narrow. As mentioned in our paper, our work operates in the high-dimensional regime, so weight_num_list should be wide enough. However, when I tried to widen this, the validation accuracy still mismatched to the Figure 3, although it did improve. Honestly, I am unsure what might be wrong with this code. It has been so long that I have forgotten the details about our codes, so I wrote a new code for NTK-LC algorithm to compress fully-connected network, and it works very well. If you are interested, you can give it a try! The new code is in the code/compression/new_MNIST/performance_MNIST.py file.

Regarding your second question, we have recently developed a trainable NTK-LC algorithm that enhances the robustness and performance of the compressed network, and we plan to share our new works as soon as possible. Additionally, there are various methods available for compressing trained networks, like APoT、N2UQ. Hope useful for you!