Lastest commit incompatible with local_attention

[MarcusLoppe]

I've haven't investigate this but the latest commit makes the MeshGPT tests fail & users get the error below:

File /usr/local/lib/python3.10/dist-packages/local_attention/transformer.py:152, in LocalMHA.forward(self, x, mask, attn_bias, cache, return_cache)
149 out = einsum(attn, v, 'b h i j, b h j d -> b h i d')
151 else:
--> 152 out = self.attn_fn(q, k, v, mask = mask, attn_bias = attn_bias)
154 if return_cache:
155 kv = torch.stack((k, v))

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)
1516 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1517 else:
-> 1518 return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1527, in Module._call_impl(self, *args, **kwargs)
1522 # If we don't have any hooks, we want to skip the rest of the logic in
1523 # this function, and just call forward.
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1525 or _global_backward_pre_hooks or _global_backward_hooks
1526 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1527 return forward_call(*args, **kwargs)
1529 try:
1530 result = None

File /usr/local/lib/python3.10/dist-packages/local_attention/local_attention.py:162, in LocalAttention.forward(self, q, k, v, mask, input_mask, attn_bias, window_size)
160 if exists(self.rel_pos):
161 pos_emb, xpos_scale = self.rel_pos(bk)
--> 162 bq, bk = self.rel_pos.apply_rotary_pos_emb(bq, bk, pos_emb, scale = xpos_scale)
164 # calculate positions for masking
166 bq_t = b_t

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1695, in Module.getattr(self, name)
1693 if name in modules:
1694 return modules[name]
-> 1695 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

AttributeError: 'SinusoidalEmbeddings' object has no attribute 'apply_rotary_pos_emb'

[lucidrains]

@MarcusLoppe oh, local attention library actually uses its own rotary embeddings

how do you reproduce that error?

[MarcusLoppe]

Fresh install of MeshGPT i think, i got a comment on the huggingface model & when i merged your latest MeshGPT commit i got the error: https://github.com/MarcusLoppe/meshgpt-pytorch/actions/runs/9763730329/job/26950298778

Edit; Your latest MeshGPT test-check got the same error

[lucidrains]

@MarcusLoppe it is some issue with beartype, i've removed it for now

thanks for letting me know!