search

imoneoi / multipack_sampler

Multipack distributed sampler for fast padding-free training of LLMs
MIT License
175 stars 13 forks source link

Algorithm does not work for n=1 #4

Open LingxiaoShawn opened 4 weeks ago

LingxiaoShawn commented 4 weeks ago

Hi author,

Thank you for the great work. The algorithm runs very fast! However, I think the current algorithm does not consider the corner case with just single GPU (n=1), and in this case, the allocate function's while loop just run forever.

Is there a way to easily fix the problem?

Thank you!

LingxiaoShawn commented 4 weeks ago

I changed line the to 

        if l <= n:
            break  # Can't allocate each sequence to a single machine

and it works, not sure whether this is correct.

LingxiaoShawn commented 1 week ago

I have fixed several bugs at https://github.com/LingxiaoShawn/multipack_sampler/blob/master/multipack_sampler.py