Open hyunwoongko opened 2 years ago
I'm not sure about the function of stop_text, could you explain a little bit or show the code?
On Mon, Jul 11, 2022 at 5:30 PM Kevin Ko @.***> wrote:
Are there some features like stop_text of faster transformer in Light seq?
— Reply to this email directly, view it on GitHub https://github.com/bytedance/lightseq/issues/337, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELIZAITJPUA76HKT43LYPLVTPSRVANCNFSM53G3IIMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
This is a simple example of stop texts feature. Let's take a dialogue with GPT model as an example.
prompt = """This is a chat between Bot and User.
Bot: Hi
User: What are you doing?
Bot:
"""
output = model.generate(prompt)
print(output)
I am talking with you.
User: Oh I see
Bot: How about you?
...(skip)
In this example, I want to stop generation when User:
is generated like the following.
prompt = """This is a chat between Bot and User.
Bot: Hi
User: What are you doing?
Bot:
"""
stop_texts = ["User:", "Bot:"]
output = model.generate(prompt, stop_texts=stop_texts)
print(output)
I am talking with you.
Faster Transformer has this feature. So I am wondering Lightseq has this feature or any plan. FYI, I also made a library like this that can be used with HuggingFace Transformers. https://github.com/hyunwoongko/stop-sequencer
Thanks for your explanation. We are working on refactoring the searching module and will consider this feature. Currently, we can only let lightseq infer the whole sentence and do some post-processing.
On Sat, Jul 16, 2022 at 12:22 PM Kevin Ko @.***> wrote:
This is a simple example of stop texts feature. Let's take a dialogue with GPT model as an example.
prompt = """This is a chat between Bot and User.Bot: HiUser: What are you doing?Bot:""" output = model.generate(prompt)print(output)
- output
I am talking with you. User: Oh I see Bot: How about you? ...(skip)
In this example, I want to stop generation when User: generated like the following.
prompt = """This is a chat between Bot and User.Bot: HiUser: What are you doing?Bot:""" stop_texts = ["User:", "Bot:"]output = model.generate(prompt, stop_texts=stop_texts)print(output)
- output
I am talking with you.
FYI, I made a library like this that can be used with HuggingFace Transformers. https://github.com/hyunwoongko/stop-sequencer
— Reply to this email directly, view it on GitHub https://github.com/bytedance/lightseq/issues/337#issuecomment-1186085785, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELIZAKGACZYZNEWHOSRYIDVUI2JDANCNFSM53G3IIMQ . You are receiving this because you commented.Message ID: @.***>
Thanks for answering :)
Are there some features like
stop_texts
orstop_ids
of faster transformer in Lightseq?for example, if I want to finish generation when
hello
token is generated, Can I stop the generation process?