TypeError: 'NoneType' object is not callable

[ThreeBoxWithCode]

Wonderful work！ When I configured the environment and gradually debugged the inference.ipynb, I got the error on Visual Prompt Decoding: Point: TypeError: 'NoneType' object is not callable. The detailed error information is as follows:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[19], line 21
     19 sem_tokens, sem_embeds = outputs["sem_tokens"], outputs["sem_embeds"]
     20 concepts, scores = model.predict_concept(sem_embeds[mask_index])
---> 21 captions = model.generate_text(sem_tokens[mask_index][:, None, :])
     23 # Display comprehensive visual understanding.
     24 text_contents = [v.flatten()[0] for v in (concepts, iou_scores, scores, captions)]

File [~/anaconda3/envs/sam/lib/python3.8/site-packages/torch/autograd/grad_mode.py:27](https://file+.vscode-resource.vscode-cdn.net/home/zhong/code/SAM/tokenize-anything/notebooks/~/anaconda3/envs/sam/lib/python3.8/site-packages/torch/autograd/grad_mode.py:27), in _DecoratorContextManager.__call__..decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File [~/anaconda3/envs/sam/lib/python3.8/site-packages/tokenize_anything/modeling/image_tokenizer.py:189](https://file+.vscode-resource.vscode-cdn.net/home/zhong/code/SAM/tokenize-anything/notebooks/~/anaconda3/envs/sam/lib/python3.8/site-packages/tokenize_anything/modeling/image_tokenizer.py:189), in ImageTokenizer.generate_text(self, visual_tokens, max_gen_len, temperature)
    187 decode_seq_len = cur_pos - prev_pos
    188 x = torch.from_numpy(tokens[:, prev_pos:cur_pos]).to(device=prompts.device)
--> 189 logits = self.text_decoder.transformer(prompts, x, prev_pos)
    190 next_logits = logits[: x.size(0), decode_seq_len - 1]
    191 if temperature > 0:

File [~/anaconda3/envs/sam/lib/python3.8/site-packages/torch/nn/modules/module.py:1194](https://file+.vscode-resource.vscode-cdn.net/home/zhong/code/SAM/tokenize-anything/notebooks/~/anaconda3/envs/sam/lib/python3.8/site-packages/torch/nn/modules/module.py:1194), in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
...
---> 73 q = apply_rotary_emb(q, cos, sin, interleaved=True, inplace=inplace)
     74 k = apply_rotary_emb(k, cos, sin, interleaved=True, inplace=inplace)
     75 return q, k

TypeError: 'NoneType' object is not callable

[PhyscalX]

Hi, @ThreeBoxWithCode, we use None fallback to handle the import error of FlashAttention2. Please check if your environment satisfies flash-attn>=2.3.3 for Ampere, Ada, or Hopper GPUs.

[ThreeBoxWithCode]

Thank you for your reply. I installed flash-attn by "pip install flash-attn --no-build-isolation" before. Now, I reinstall the flash-attn wheel, which version is 2.3.5 with cuda11.7, torch1.13.1 but it still cannot work Should I build the cilp before running the inference.ipynb?

And I test to import the flash-attn, It can work:

Python 3.8.17 (default, Jul  5 2023, 21:04:15) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from flash_attn.flash_attn_interface import (
...     flash_attn_func,
...     flash_attn_kvpacked_func,
...     flash_attn_qkvpacked_func,
...     flash_attn_varlen_func,
...     flash_attn_varlen_kvpacked_func,
...     flash_attn_varlen_qkvpacked_func,
...     flash_attn_with_kvcache,
... )
>>> 

[PhyscalX]

Test your environment with the following imports:

from flash_attn import flash_attn_func
from flash_attn import flash_attn_with_kvcache
from flash_attn.layers.rotary import apply_rotary_emb

Reference: https://github.com/baaivision/tokenize-anything/blob/main/tokenize_anything/modeling/text_decoder.py

[ThreeBoxWithCode]

They all work.

[PhyscalX]

If it works, the apply_rotary_emb should not be None, restart your Jupyter kernel

[ThreeBoxWithCode]

Yes. When I restarted the Jupyter kernel, it worked. It seems that installing flash-attn with pip install flash-attn --no-build-isolation does not work Thank you for answering my questions late at night.

[PhyscalX]

Pip works. But VSCode Jupyter Extension will not reload python runtime until next restarting 😅.