search

pytorch / opacus

Training PyTorch models with differential privacy
https://opacus.ai
Apache License 2.0
1.65k stars 328 forks source link

Opacus on cifar10 attempts to broadcast tensors in an incorrect way #607

Closed jon-chuang closed 8 months ago

jon-chuang commented 8 months ago

🐛 Bug

When tracing with dynamo with proper support for comparisons

Failed running call_function <built-in function eq>(*(Parameter(FakeTensor(..., device='cuda:0', size=(10, 512), dtype=torch.bfloat16,
           requires_grad=True)), Parameter(FakeTensor(..., device='cuda:0', size=(10,), dtype=torch.bfloat16,
           requires_grad=True))), **{}):
Attempting to broadcast a dimension of length 10 at -1! Mismatching argument at index 1 had torch.Size([10]); but expected shape should be broadcastable to [10, 512]

Full logs

One should never attempt to broadcast a tensor of shape(x) to one of shape(x, y). There needs to be an unsqueeze(-1) on the first arg

To Reproduce

Hard to provide a reproducer for now (will investigate further and update). See: https://github.com/pytorch/pytorch/pull/111492

jon-chuang commented 8 months ago

I've isolated it to a dynamo-only issue: https://github.com/pytorch/pytorch/issues/111542