search

shibing624 / textgen

TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLOOM，GPT2，Seq2Seq，BART，T5，UDA等模型的训练和预测，开箱即用。
Apache License 2.0
937 stars 109 forks source link

请问有没有用于evaluation的代码？ #26

Closed xiongxiaochu closed 1 year ago

xiongxiaochu commented 1 year ago

代码里只有train和predict的代码，尝试写了一下evaluate的代码，但是总是不对，train的时候读数据只有input_ids，是不是evaluate的时候要加上labels？之前参照原始chatglm的代码修改了一下，也总是报错，请问作者大大能不能给一下evaluate的代码呀？

shibing624 commented 1 year ago

eval的指标要哪个？ loss吗，还是bleu,rouge_l 吗？

xiongxiaochu commented 1 year ago

eval的指标要哪个？ loss吗，还是bleu,rouge_l 吗？

都需要

xiongxiaochu commented 1 year ago

eval的指标要哪个？ loss吗，还是bleu,rouge_l 吗？

Traceback (most recent call last): File "/media/cfs/zhangzhiqian/chatgpt/chatglm/lora/evaluate_v2.py", line 290, in main() File "/media/cfs/zhangzhiqian/chatgpt/chatglm/lora/evaluate_v2.py", line 269, in main trainer = FinetuneTrainer( File "/opt/conda/envs/chatglm/lib/python3.9/site-packages/transformers/trainer.py", line 491, in init self._move_model_to_device(model, args.device) File "/opt/conda/envs/chatglm/lib/python3.9/site-packages/transformers/trainer.py", line 733, in _move_model_to_device model = model.to(device) AttributeError: 'ChatGlmModel' object has no attribute 'to'

我加载了包含lora的模型之后，初始化trainer的时候会报这个错，求问是什么问题呀？

shibing624 commented 1 year ago

就提示的错误，解决就好，不加model.to(device), 你evaluate_v2.py", line 269 写了啥

xiongxiaochu commented 1 year ago

就提示的错误，解决就好，不加model.to(device), 你evaluate_v2.py", line 269 写了啥

trainer = FinetuneTrainer(
        model=model,
        train_dataset=eval_dataset,
        eval_dataset=eval_dataset,
        args=training_args,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics
    )

就是初始化trainer，但是那个model.to(device)是在transformer的trainer里，如何修改？求问大大eval的代码现在有了么？

xiongxiaochu commented 1 year ago

就提示的错误，解决就好，不加model.to(device), 你evaluate_v2.py", line 269 写了啥

我用了chatglm的ptuning里的trainer放到本地，然后强制不执行这个代码，但是后面报了别的错，

image

这是啥原因啊😭

shibing624 commented 1 year ago

added: https://github.com/shibing624/textgen/commit/2eb60a295a06435c1221034a3d17b6f20740a82b#diff-4a3a3b4cf157712b036a7ff3c50a1a7c8c8f6032a3a175a5a372ab5fa8d42a1b eval_loss

xiongxiaochu commented 1 year ago

added: 2eb60a2#diff-4a3a3b4cf157712b036a7ff3c50a1a7c8c8f6032a3a175a5a372ab5fa8d42a1b eval_loss

好的，感谢！

xiongxiaochu commented 1 year ago

added: 2eb60a2#diff-4a3a3b4cf157712b036a7ff3c50a1a7c8c8f6032a3a175a5a372ab5fa8d42a1b eval_loss

请问下，如果我想加载训练好的lora模型，直接evaluate，要怎么写呀？

shibing624 commented 1 year ago

tran_data = 10条，全放入eval_data就行了，跑0.1个epoch

xiongxiaochu commented 1 year ago

tran_data = 10条，全放入eval_data就行了，跑0.1个epoch

就是我加载训练好的模型，然后do_train，train_data里放10条数据，需要eval的数据放到eval_data里传入train_model()方法里是吗？

shibing624 commented 1 year ago

en