Closed SD325 closed 2 months ago
Hi, currently gradient computation with multiple GPUs is not supported mostly because I have not found a way to use vmap
to calculate gradients for each instance within a batch. If you'd like to use multiple GPUs, you can split your data into multiple files and launch a separate job for each subset of data.
Hi, If I launch separate jobs, will the projection matrix remain the same when I set a same number for model_id?
Yes! It will
How can we utilize multiple GPUs for the gradient feature collection step? The current implementation only works with a single GPU.