How can I get multi results from once inference ?

[MowanHu]

Thank you very much for such a great work ！ At present, I have successfully converted the bart model and beam search decoding strategy into lightseq, but each inference can only return one result, how can I get multiple beam results?

期待回复，感谢！

[godweiyang]

The results of LightSeq Transformer model should include all the beam results. In the following line, you can print ls_res_ids directly to see the complete results. https://github.com/bytedance/lightseq/blob/master/examples/inference/python/test/ls_bart.py#L31

[MowanHu]

The results of LightSeq Transformer model should include all the beam results. In the following line, you can print ls_res_ids directly to see the complete results. https://github.com/bytedance/lightseq/blob/master/examples/inference/python/test/ls_bart.py#L31

感谢韦神的回复！目前还是有点问题

我使用的环境版本:
lightseq==2.2.0.post1

模型转换参数： generator config beam size: 8 extra decode length(max decode length - src input length): 30 length penalty: 1 diverse lambda: 0 sampling method: beam_search

仿照ls_bart.py执行代码： title_tensor = torch.tensor(title_idx) predict_output = model.infer(title_tensor) print(predict_output)

(array([[[1395, 800, 1469, 2478, 727, 4415, 1825, 4794, 4761, 6399, 102]]], dtype=int32), array([[-0.64969325]], dtype=float32))

打印出predict_output，还是只能看到一条结果，并没有8个结果，是我还有哪里参数没有配置吗？ 另外，后面这个array中的-0.64969325是beamscore吗？

[Taka152]

Could you try to add multiple_output=True in the infer function?

[MowanHu]

thx, it works ! and where can I view the usage description of the infer function?

---

print(predict_output[1]) array([[-0.614843 , -0.7758462, -0.9982237, -1.038495 , -1.0539751,-1.3415337, -1.3432366, -1.4289901]], dtype=float32))

Is the element in the array equal to beam score by using log-softmax to calulate? It seems low confidence, 0.5407257825933576, 0.46031409835746295, 0.36853348614086223, 0.3539870317478623, 0.3485494726615821, 0.261444383682408, 0.26099954890350463, 0.2395507224007668

[Taka152]

You can try add multiple_output=True parameter in infer function.

On Tue, Jul 12, 2022 at 5:26 PM 冷墙 @.***> wrote:

Thank you very much for such a great work ！ At present, I have successfully converted the bart model and beam search decoding strategy into lightseq, but each inference can only return one result, how can I get multiple beam results?

期待回复，感谢！

— Reply to this email directly, view it on GitHub https://github.com/bytedance/lightseq/issues/338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELIZAPVBAHPJIXL5LM5RNTVTU23HANCNFSM53KKJYFQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>