search

eml-eda / mixprec-pruning

3 stars 0 forks source link

The issue of model evaluation results #3

Open tang12138-yyhh opened 21 hours ago

tang12138-yyhh commented 21 hours ago

  1. Hello, I would like to ask about the evaluation of the results after the quantization and pruning (icl_mixprec_training_cost_size.py) has been completed, which has generated .pt files recorded in the three stages (warmup, search, and finetuning stage), as well as .ckp files for the three stages (e.g. final_best_search.ckp, final_best_finetuning.ckp, final_best-warmup.ckp). I would like to know how to evaluate the results and compare them with the results of quantization and pruning performed separately or in sequence. Is there any parameter that can make the results more visual? thanks.

  2. I only ran the file (icl_mixprec_training_cost_size.py) once, and the result was .pt and .ckp files. Is there something wrong with my process?

bmots commented 13 hours ago

Hello @tang12138-yyhh,

  1. In the log file that is created by the script you can find the regularization metrics of interest (e.g. size of the DNN, number of inference cycles, energy) and the accuracy on the benchmark. To compare the results of the joint methodology with the sequential application of pruning and MPQ you can create, for instance, a plot showing the Pareto front of the obtained architectures in the cost (size or a hardware-specific metric) vs accuracy space, as in Figure 5 of our paper. You can additionally evaluate the results in terms of required training time, as explained in Sec. 5.3 of the paper.
  2. To obtain an entire Pareto front of architectures, you have to run the script multiple times varying the regularization strength hyperparameter, that controls the trade-off between the task loss and regularization loss terms. You can do this by regulating the strength argument in the script.

I hope this answers your questions.