Closed rishabluthra closed 10 months ago
The reason is, as pointed out in https://github.com/MadryLab/trak/issues/44, that grads
gets initialized with the dtype
of batch[0]
:
grads = torch.empty(size=(batch[0].shape[0], self.num_params),
dtype=batch[0].dtype,
device=batch[0].device)
Instead, it should be initialized as float16
/float32
.
Fixed in https://github.com/MadryLab/trak/commit/fb5e0a495a859359b382b6b2a066800f5f7f66c7 (on branch https://github.com/MadryLab/trak/tree/0.2.2 for now, will be merged in main in the next release)
Hey, I am trying out the BERT QNLI example given in the repo on a A100 on Colab and keep running into this Runtime issue when trying to featurize
during this code sample:
It seems the batch, when, processed is returned as ints so I end up modifying the process function as
but that still has the same issue
Was just wondering if you ever ran into the issue when trying it out on QNLI or if I am missing anything, thanks!