tuidan
commented
2 months ago Hi, thanks for your questions!
- Generally, it is common that the compressed models generate repeated letters or phrases, especially when the compression ratio is growing larger, which has also been mentioned in previous works such as LLM-Pruner.
- During our experiments, we also encountered this phenomenon and we found SVD-LLM is much more robust than other baseline SVD compression methods, which means that SVD-LLM could still ensure a good generation at the compression ratio where other baselines totally lose their generation ability.
- One of the useful tricks to avoid generating random and repetitive outputs is to adjust the temperature for generation. Another is to fine-tune the compressed LLM following the guidelines in our repo. Fine-tuning is also a good and popular method to enhance the compressed models' generation ability and help us achieve a higher compression ratio.
- Regarding the compressed weights: We are planning to share the compressed weights after our paper is accepted by the conference.
Description When running the code, we successfully obtain a compressed model. However, when prompted with an input, the model generates random and repetitive outputs, often repeating the same letters or phrases. This significantly impacts the usability and reliability of the model.
Request What could be the reason for this issue? Could you please share your compressed weights to help diagnose and resolve the problem?
Thank you!