search

pytorch / ao

The missing pytorch dtype and layout library for training and inference
BSD 3-Clause "New" or "Revised" License
458 stars 59 forks source link

[Question] MBU in automated CI? #237

Open cadedaniel opened 2 months ago

cadedaniel commented 2 months ago

Hi folks, thanks for the great work.

With https://github.com/pytorch/ao/pull/135 merged, vLLM could see benefit from torch.compile backend given compiler-native integration with PagedAttention kernels.

Is there an easy way to see what the latest/nightly MBU is for torch compile on say, H100 / Llama3 70B?

Also interested in cold start compile time

cc @msaroufim

supriyar commented 2 months ago

@anijain2305 do we have any benchmark numbers for the cold start compile time?

msaroufim commented 2 months ago

Related https://github.com/pytorch/pytorch/issues/125958