search

LambdaLabsML / distributed-training-guide

Best practices & guides on how to write distributed pytorch training code
8 stars 0 forks source link

Add tensor parallelism to 405b chapter or advanced topics? #37

Open corey-lambda opened 3 days ago

corey-lambda commented 3 days ago

See https://pytorch.org/docs/stable/distributed.tensor.parallel.html

llama 405b paper discusses using FSDP, pipeline parallelism, context parallelism, and tensor parallelism

It'd be relatively straightforward to add tensor parallelism to specific modules (Linear/Embedding)

corey-lambda commented 1 day ago

Adding this in #39