search

OpenNMT / CTranslate2

Fast inference engine for Transformer models
https://opennmt.net/CTranslate2
MIT License
3.37k stars 299 forks source link

Support soft prompt #1117

Open cccntu opened 1 year ago

cccntu commented 1 year ago

Hi,

I would like to use CTranslate2 to accelerate BLIP2, which uses a soft prompt to generate text with OPT model. https://github.com/salesforce/LAVIS/tree/main/projects/blip2

But I think CTranslate2 is lacking support for soft prompt. It only accepts start_tokens. https://opennmt.net/CTranslate2/python/ctranslate2.Generator.html

guillaumekln commented 1 year ago

Hi,

More generally CTranslate2 does not support the BLIP2 architecture. In fact it does not support any vision models (see also #979).

Currently there is no plan to support these models.

cccntu commented 1 year ago

Hi, Thanks for your reply.

How about adding support for using soft prompt? I.e. instead of only allowing passing in a list of tokens, user can pass in a list of token embedding.

- generate_batch(start_tokens: List[List[Union[str]], *, 
+ generate_batch(start_tokens: List[List[Union[str, List[Float]]], *, 
...)

I don't need a full C++ implementation for blip2, I think I just need this feature to accelerate OPT in bilp2. Thanks.