search

HigherOrderCO / Bend

A massively parallel, high-level programming language
https://higherorderco.com
Apache License 2.0
16.67k stars 410 forks source link

Request: Multi GPU parallelism #358

Open Jainam2130 opened 1 month ago

Jainam2130 commented 1 month ago

Does this natively support parallelism accross gpus? Also feature request: natively perform flash attention please

developedby commented 1 month ago

It doesn't support working across multi gpus yet. First we would require a 64bit implementation of HVM to be able to use all resources and then a significant change to the cuda runtime to be able to use multiple GPUS. An interesting idea to pursue in the not-so-far future, but not our immediate priority.

dxvid-pts commented 1 month ago

Also does this support parallelism across the whole system (eg across CPU + multiple GPUs)?