search

project-baize / baize-chatbot

Let ChatGPT teach your own chatbot in hours with a single GPU!
https://arxiv.org/abs/2304.01196
GNU General Public License v3.0
3.15k stars 275 forks source link

Why not use huggingface generate() and write custom greedy search? #12

Closed Moses0h closed 1 year ago

Moses0h commented 1 year ago

I'm trying to run batch generation but greedy_search() seems to work only for single input_ids. Curious why you guys implemented greedy_search()?

guoday commented 1 year ago

For demo, if you run batch generation, you will need to wait for inference to complete so the user experience is not good. You can use this code for batch generation.

JetRunner commented 1 year ago

Closing it for now and will add batch inference option in our to-do list.

Moses0h commented 1 year ago

writing here because related,

I'm currently using 0 as padding token (correct me if I'm wrong), left-padding, padding="max_length" However, the more the padding, the generated output becomes more and more weird.

for example, inputs_ids [0,0,0,0,..... 23,5,143,24, etc...] has weird generated outputs and just [23, 5, 143, 24, etc...] works normally.