Distributed inference issue

dvlab-research / LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

http://arxiv.org/abs/2309.12307

Apache License 2.0

2.62k stars 274 forks source link

Distributed inference issue #171

Open yixliu1 opened 9 months ago

yixliu1 commented 9 months ago

Hi there,

I found out model is distributed loaded on each gpus while inferencing, but for each time iteration, only one data sample is being infereced. Is there anyway that we can dealing with multiple data samples at the same time?