search

mikeybellissimo / LoRA-MPT

A repo for finetuning MPT using LoRA. It is currently configured to work with the Alpaca dataset from Stanford but can easily be adapted to use another.
Apache License 2.0
18 stars 7 forks source link

Problem about generate #3

Open jianchaoji opened 1 year ago

jianchaoji commented 1 year ago

Thanks for the repo!

I used your code to trained the model. But when I tried to do the inference, the output is just the copy of the input. Have you met this problem before?

mikeybellissimo commented 1 year ago

Hi you’re more than welcome, happy to hear you were able to train and sorry to hear you’re having trouble with inference! I have not encountered that issue personally but lets see if we can work it out.

Are you using the generate.py script for inference?

Any additional details you’re able to provide about what how to replicate the error would be very helpful as well.

jianchaoji commented 1 year ago

Thank you for your response! Yes, I'm using generate.py script for inference. And for the training, I just changed the base model path, because I downloaded the mpt-7b.

mikeybellissimo commented 1 year ago

That is very strange. You’re going through the gradio interface it sets up?

I've actually only really used the mpt-7b-instruct version so far so that might be a source of the error itself. The pretrained, non fine tuned model (non-instruct) just does next word prediction pretty much without knowledge of how to follow instructions/answer questions so if your fine tuning isn't sufficient to teach it this process then it could potentially cause this error but I would still expect it to at least generate some different text/more text. Im about to go to bed but can try and briefly look at this tomorrow morning and look in detail on Monday if you’re still having issue.

jianchaoji commented 1 year ago

No problem. I will do more tests, and I will let you know if I still have that problem. Thank you!

jianchaoji commented 1 year ago

Now for the original dataset, it can be fine tuned well. But if I changed it to my self constructed dataset, it still have the same error (Repeat the input). Have you tried your customized dataset?

mikeybellissimo commented 1 year ago

Yeah it worked fine. Did you continue to use the Seq2Seq collator and set everything up in the same format as the Alpaca in terms of compatibility with the prompt generator and everything?

mikeybellissimo commented 1 year ago

I tested with the vanilla Mpt model and had no issue either. Ill be around tomorrow if you’re still having issue I can make time for a video call or something if you’d like. I learn from working through these things as well so its no bother.

jianchaoji commented 1 year ago

I changed nothing but the dataset format. I set everything up in the same format as the Alpaca. And I used the same dataset for llama-lora and it worked fine.

Yeah, it would be good if we can make time for a video call on zoom tomorrow.

jianchaoji commented 1 year ago

If you don't mind, could you please share your customized dataset? So I can double check the right format.

mikeybellissimo commented 1 year ago

Sure, I don't see a way to do a private message on github. I made a burner email so I don't get my real one loaded with spam. Feel free to reach out to plan the zoom, I can late-afternoon/night today or tomorrow. Email: githubtemporarytemp@gmail.com

jianchaoji commented 1 year ago

Sorry for the late reply. I sent you an email a few days ago, but I'm not sure I use the burner email correctly. I think I find the where the problem is. I will let you know if I have any updates and problem. Thank you so much for your help.