'SinusoidalEmbeddings' object has no attribute 'apply_rotary_pos_emb'

[MarcusLoppe]

During the one of recent commits, the apply_rotary_pos_emb function is expected to exists within the positioning embedding class but since SinusoidalEmbeddings is outside of the class/object it can't be found. When using version 1.9.1 of local attention this error doesn't show up.

I'd recommend moving the apply_rotary_pos_emb so it's within the SinusoidalEmbeddings class.

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)
1516 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1517 else:
-> 1518 return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1527, in Module._call_impl(self, *args, **kwargs)
1522 # If we don't have any hooks, we want to skip the rest of the logic in
1523 # this function, and just call forward.
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1525 or _global_backward_pre_hooks or _global_backward_hooks
1526 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1527 return forward_call(*args, **kwargs)
1529 try:
1530 result = None

File /usr/local/lib/python3.10/dist-packages/local_attention/local_attention.py:162, in LocalAttention.forward(self, q, k, v, mask, input_mask, attn_bias, window_size)
160 if exists(self.rel_pos):
161 pos_emb, xpos_scale = self.rel_pos(bk)
--> 162 bq, bk = self.rel_pos.apply_rotary_pos_emb(bq, bk, pos_emb, scale = xpos_scale)
164 # calculate positions for masking
166 bq_t = b_t

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1695, in Module.getattr(self, name)
1693 if name in modules:
1694 return modules[name]
-> 1695 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

AttributeError: 'SinusoidalEmbeddings' object has no attribute 'apply_rotary_pos_emb'

[lucidrains]

@MarcusLoppe hello again 😄 not sure what's going on, but i just decided to revert to importing the apply_rotary_pos_emb function

let me know if that does it!

[MarcusLoppe]

@MarcusLoppe hello again 😄 not sure what's going on, but i just decided to revert to importing the apply_rotary_pos_emb function

let me know if that does it!

Hiya :)

It's very strange, it didn't work :/

This is probably some python thing with the initializations of the class or what not. Maybe there is some edge case where referencing the function using self.apply_rotary_pos_emb isn't working and must be a class object or what not. Maybe some CUDA hook or whatever is not picking it up? :/

If I run the code outside of MeshGPT it works since it's throwing me a error.

from local_attention.rotary import SinusoidalEmbeddings
test = SinusoidalEmbeddings(dim = 192)
test.apply_rotary_pos_emb()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 3
      1 from local_attention.rotary import SinusoidalEmbeddings
      2 te = SinusoidalEmbeddings(dim = 192)
----> 3 te.apply_rotary_pos_emb()

File /opt/conda/lib/python3.10/site-packages/torch/amp/autocast_mode.py:16, in autocast_decorator.<locals>.decorate_autocast(*args, **kwargs)
     13 @functools.wraps(func)
     14 def decorate_autocast(*args, **kwargs):
     15     with autocast_instance:
---> 16         return func(*args, **kwargs)

TypeError: apply_rotary_pos_emb() missing 3 required positional arguments: 'q', 'k', and 'freqs'

[lucidrains]

@MarcusLoppe ok, that is strange, want to try yet again?

[MarcusLoppe]

@MarcusLoppe ok, that is strange, want to try yet again?

Nice, all good now. Thank you :) 🚀

[MarcusLoppe]

@lucidrains

Oh, i just noticed this error when trying to create MeshTransformer.

File /opt/conda/lib/python3.10/site-packages/pytorch_custom_utils/save_load.py:32, in save_load.<locals>._save_load.<locals>.__init__(self, *args, **kwargs)
     30 @wraps(_orig_init)
     31 def __init__(self, *args, **kwargs):
---> 32     _config = pickle.dumps((args, kwargs))
     34     setattr(self, config_instance_var_name, _config)
     35     _orig_init(self, *args, **kwargs)

TypeError: cannot pickle 'staticmethod' object

[lucidrains]

@MarcusLoppe ah ok, i'll just remove the staticmethod altogether, thanks!