Question regarding usage of generate when input_embeds is present

[soumyasanyal]

Thanks for sharing the code! We have been trying to integrate this into our project. One challenge we find is that our use-case needs us to pass some input_embeds to the generate function, instead of input_ids. Is there a plan to support that in the code-base or can you help us in understanding the necessary files that need to be changed to support inputs_embeds as an input option?

For reference, the Huggingface generate function has inputs that can be both ids and embeddings.

Thanks in advance!

[chansonzhang]

I found a description in Huggingface's generation function parameter desciption.

For decoder-only models inputs should of in the format of input_ids.

I guess perhaps the neurologic_decoding is a decoder-only model so the inputs can only be input_ids.

[GloriaXimingLu]

Hi, thanks for the question! Our code is an extension of generation function in huggingface, so we automatically support all their parameters. You could pass in input_embeds in our generate function in the same way as using model.generate function in huggingface.