search

thunlp / Ouroboros

Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
Apache License 2.0
77 stars 9 forks source link

Tensor Parallelism #8

Open lethean1 opened 1 month ago

lethean1 commented 1 month ago

I want to use tensor parallelism with ouroboros, but I do not find the config to start the tensor parallel, can you give me an example?

Achazwl commented 1 month ago

Our code is based on huggingface transformers. Currently, it only supports pipeline parallelism. Tensor Parallel may need extra modification, such as implementing based on gpt-fast.