bmots
commented
10 hours ago Hello @tang12138-yyhh,
each of those scripts can be used to train a model with a different technique, that I will detail below:
icl_mixprec_training_cost_size.py: this script allows to train a model with the proposed Mixed-Precision Quantization (MPQ) + pruning approach. To do so, you can set the argumentsearch-typeto "channel", and include 0 in the candidate weights precisions set, with the argumentw_precisions. The regularization objective used within this script is the size (the default one), i.e. the number of parameters of the neural network.icl_mixprec_training_cost_general.py: this script is similar to the previous one but uses a different regularizer, that can be chosen among the MPIC and the NE16 ones, described in our paper. To do so, you can set the argumenthardware-modelto either "mpic" or "ne16". More regularizers are available however in the new version of the library, feel free to check it out!icl_PIT_training.py: this script allows you to train a network with PIT, the structured pruning algorithm that we compare with in the paper. More details on PIT can be found in this paper. This method applies structured pruning only, and does not quantize the model. It can be used to evaluate the sequential application of a pruning and a MPQ algorithm.icl_PIT+mixprec_training_cost_size.py: this script applies a channel-wise MPQ algorithm on top of the output of PIT, the structured pruning algorithm mentioned in the previous bullet point. Thus, one can use the scripticl_PIT_training.pyto obtain pruned architectures to be used as input for the MPQ approach with theicl_PIT+mixprec_training_cost_size.pyscript. For this reason some input checkpoints (the ones that you mention) are required here. You can obtain them by runningicl_PIT_training.py(also multiple times to obtain a Pareto-front of pruned architectures, then you can choose the seed architecture within the obtained ones and pass their path as argument topit_checkpointin theicl_PIT+mixprec_training_cost_size.pyscript). Two checkpoints are used in the script to first load the architecture in the PIT format implemented within the PLiNIO library, then here the optimized architecture is exported and finally the fine-tuned weights are loaded here. The loading of the two separate checkpoints could also be avoided by calling thetorch.save()function directly on the PIT model after its fine-tuning phase.
The proposed approach is also available in the new version of the PLiNIO library, with more regularizers and a more user-friendly structure. If you are interested, you can have a look here!
The other files in the repository follow a similar structure, but refer to different benchmarks. The aforementioned ones consider CIFAR-10, the kws scripts consider the Google Speech Commands v2 dataset, the tin ones Tiny Imagenet and the imn one uses the ImageNet dataset.
Let us know if you have any further questions.
How are the relationships between these codes, such as icl_PIT+mixprec_training_cost_size.py, icl_PIT_training.py, icl_mixprec_training_cost_size.py, and icl_mixprec_training_cost_general.py? What do each of these four files implement, and in icl_PIT+mixprec_training_cost_size.py, I see that the final_best_finetuning.ckp and final_best_search.ckp files are used, but when I run it, I find that these two files are missing. I wonder where these files came from?