search

PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
https://paddlenlp.readthedocs.io
Apache License 2.0
12.1k stars 2.94k forks source link

Trainer Compress API evaluation and return arguments #3747

Closed sijunhe closed 1 year ago

sijunhe commented 1 year ago

Feature request

当前似乎只有dynabert有evaluation, 但是后续的ptq, qat都没有. 能否考虑在所有策略执行完毕后进行统一的evaluation, 然后将关键信息(metrics, export model path, model size (MACs or memory footprint)) log下来并且作为return arguments返回？

Motivation

trainer.compress应该与trainer.train有类似的API. trainer.train会返还一个TrainOutput, 同时每一次evaluation都有UI显示.

Your contribution

n/a

LiuChiachi commented 1 year ago

感谢关注和提出的建议～

目前dynabert和 qat在训练的过程中以及训练结束有evaluation信息，目前ptq还没有，之后会增加对伪量化模型（静态图）的evaluation，并且补充出更多的模型相关信息。