Stop text feature for inference module

[hyunwoongko]

Are there some features like stop_texts or stop_ids of faster transformer in Lightseq?

for example, if I want to finish generation when hello token is generated, Can I stop the generation process?

[Taka152]

I'm not sure about the function of stop_text, could you explain a little bit or show the code?

On Mon, Jul 11, 2022 at 5:30 PM Kevin Ko @.***> wrote:

Are there some features like stop_text of faster transformer in Light seq?

— Reply to this email directly, view it on GitHub https://github.com/bytedance/lightseq/issues/337, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELIZAITJPUA76HKT43LYPLVTPSRVANCNFSM53G3IIMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[hyunwoongko]

This is a simple example of stop texts feature. Let's take a dialogue with GPT model as an example.

prompt = """This is a chat between Bot and User.
Bot: Hi
User: What are you doing?
Bot:
"""

output = model.generate(prompt)
print(output)

• output

  I am talking with you.
  User: Oh I see
  Bot: How about you?
  ...(skip)

In this example, I want to stop generation when User: is generated like the following.

prompt = """This is a chat between Bot and User.
Bot: Hi
User: What are you doing?
Bot:
"""

stop_texts = ["User:", "Bot:"]
output = model.generate(prompt, stop_texts=stop_texts)
print(output)

• output

  I am talking with you.

Faster Transformer has this feature. So I am wondering Lightseq has this feature or any plan. FYI, I also made a library like this that can be used with HuggingFace Transformers. https://github.com/hyunwoongko/stop-sequencer

[Taka152]

Thanks for your explanation. We are working on refactoring the searching module and will consider this feature. Currently, we can only let lightseq infer the whole sentence and do some post-processing.

On Sat, Jul 16, 2022 at 12:22 PM Kevin Ko @.***> wrote:

This is a simple example of stop texts feature. Let's take a dialogue with GPT model as an example.

prompt = """This is a chat between Bot and User.Bot: HiUser: What are you doing?Bot:""" output = model.generate(prompt)print(output)

• output

I am talking with you. User: Oh I see Bot: How about you? ...(skip)

In this example, I want to stop generation when User: generated like the following.

prompt = """This is a chat between Bot and User.Bot: HiUser: What are you doing?Bot:""" stop_texts = ["User:", "Bot:"]output = model.generate(prompt, stop_texts=stop_texts)print(output)

• output

I am talking with you.

FYI, I made a library like this that can be used with HuggingFace Transformers. https://github.com/hyunwoongko/stop-sequencer

— Reply to this email directly, view it on GitHub https://github.com/bytedance/lightseq/issues/337#issuecomment-1186085785, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELIZAKGACZYZNEWHOSRYIDVUI2JDANCNFSM53G3IIMQ . You are receiving this because you commented.Message ID: @.***>

[hyunwoongko]

Thanks for answering :)