cvlab-columbia / viper

Code for the paper "ViperGPT: Visual Inference via Python Execution for Reasoning"
Other
1.65k stars 115 forks source link

TypeError: 'type' object is not subscriptable on running main_simple.ipynb #10

Closed earakely closed 1 year ago

earakely commented 1 year ago

Hi,

I followed the setup in the README.md. I am getting the following error when running the first cell of the main_simple.ipynb notebook:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 from main_simple_lib import *

File ~/viper/main_simple_lib.py:32
     29 cache = Memory('cache/' if config.use_cache else None, verbose=0)
     31 mp.set_start_method('spawn', force=True)
---> 32 from vision_processes import forward, finish_all_consumers  # This import loads all the models. May take a while
     33 from image_patch import *
     34 from video_segment import *

File ~/viper/vision_processes.py:21
     17 console = Console(highlight=False)
     19 if mp.current_process().name == 'MainProcess':
     20     # No need to initialize the models inside each process
---> 21     import vision_models
     22     # Create a list of all the defined models
     23     list_models = [m[1] for m in inspect.getmembers(vision_models, inspect.isclass)
     24                    if vision_models.BaseModel in m[1].__bases__]

File ~/viper/vision_models.py:144
    140         to_return = to_return.cpu()
    141         return to_return  # To save: plt.imsave(path_save, prediction.cpu().numpy())
--> 144 class CLIPModel(BaseModel):
    145     name = 'clip'
    147     def __init__(self, gpu_number=0, version="ViT-L/14@336px"):  # @336px

File ~/viper/vision_models.py:232, in CLIPModel()
    227     negative_text_features = F.normalize(negative_text_features, dim=-1)
    229     return negative_text_features
    231 @torch.no_grad()
--> 232 def classify(self, image: Union[torch.Tensor, list], categories: list[str], return_index=True):
    233     is_list = isinstance(image, list)
    234     if is_list:

TypeError: 'type' object is not subscriptable
earakely commented 1 year ago

It was due to using python version < 3.9.

Getting this repo to work is like pulling one's teeth out. There is a lot of documentation on prerequisites that is missing.

I'm now getting the below error using the xl model:

DepthEstimation loaded 
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
│                                                                                                  │
│ ❱ 1 from main_simple_lib import *                                                                │
│   2                                                                                              │
│                                                                                                  │
│ /home/ec2-user/viper2/main_simple_lib.py:32 in <module>                                          │
│                                                                                                  │
│    29 cache = Memory('cache/' if config.use_cache else None, verbose=0)                          │
│    30                                                                                            │
│    31 mp.set_start_method('spawn', force=True)                                                   │
│ ❱  32 from vision_processes import forward, finish_all_consumers  # This import loads all the    │
│    33 from image_patch import *                                                                  │
│    34 from video_segment import *                                                                │
│    35 from datasets.dataset import MyDataset                                                     │
│                                                                                                  │
│ /home/ec2-user/viper2/vision_processes.py:177 in <module>                                        │
│                                                                                                  │
│   174 │   for model_class_ in list_models:                                                       │
│   175 │   │   for process_name_ in model_class_.list_processes():                                │
│   176 │   │   │   if process_name_ in config.load_models and config.load_models[process_name_]   │
│ ❱ 177 │   │   │   │   consumers[process_name_] = make_fn(model_class_, process_name_, counter_   │
│   178 │   │   │   │   counter_ += 1                                                              │
│   179 │                                                                                          │
│   180 │   queues_in = None                                                                       │
│                                                                                                  │
│ /home/ec2-user/viper2/vision_processes.py:43 in make_fn                                          │
│                                                                                                  │
│    40 │   num_gpus = torch.cuda.device_count()                                                   │
│    41 │   gpu_number = counter % num_gpus                                                        │
│    42 │                                                                                          │
│ ❱  43 │   model_instance = model_class(gpu_number=gpu_number)                                    │
│    44 │                                                                                          │
│    45 │   def _function(*args, **kwargs):                                                        │
│    46 │   │   if process_name != model_class.name:                                               │
│                                                                                                  │
│ /home/ec2-user/viper2/vision_models.py:416 in __init__                                           │
│                                                                                                  │
│    413 │   │   BaseModel.__init__(self, gpu_number)                                              │
│    414 │   │                                                                                     │
│    415 │   │   with contextlib.redirect_stderr(open(os.devnull, "w")):  # Do not print nltk_dat  │
│ ❱  416 │   │   │   from maskrcnn_benchmark.engine.predictor_glip import GLIPDemo, to_image_list  │
│    417 │   │   │   │   create_positive_map_label_to_token_from_positive_map                      │
│    418 │   │                                                                                     │
│    419 │   │   working_dir = f'{config.path_pretrained_models}/GLIP/'                            │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/engine/predictor_glip.py:11 in <module>            │
│                                                                                                  │
│     8 from transformers import AutoTokenizer                                                     │
│     9 from torchvision import transforms as T                                                    │
│    10 import pdb                                                                                 │
│ ❱  11 from maskrcnn_benchmark.modeling.detector import build_detection_model                     │
│    12 from maskrcnn_benchmark.utils.checkpoint import DetectronCheckpointer                      │
│    13 from maskrcnn_benchmark.structures.image_list import to_image_list                         │
│    14 from maskrcnn_benchmark.structures.boxlist_ops import boxlist_iou                          │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/modeling/detector/__init__.py:1 in <module>        │
│                                                                                                  │
│ ❱  1 from .generalized_rcnn import GeneralizedRCNN                                               │
│    2 from .generalized_vl_rcnn import GeneralizedVLRCNN                                          │
│    3                                                                                             │
│    4 _DETECTION_META_ARCHITECTURES = {"GeneralizedRCNN": GeneralizedRCNN,                        │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py:11 in        │
│ <module>                                                                                         │
│                                                                                                  │
│     8                                                                                            │
│     9 from maskrcnn_benchmark.structures.image_list import to_image_list                         │
│    10                                                                                            │
│ ❱  11 from ..backbone import build_backbone                                                      │
│    12 from ..rpn import build_rpn                                                                │
│    13 from ..roi_heads import build_roi_heads                                                    │
│    14                                                                                            │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/modeling/backbone/__init__.py:6 in <module>        │
│                                                                                                  │
│     3 from torch import nn                                                                       │
│     4                                                                                            │
│     5 from maskrcnn_benchmark.modeling import registry                                           │
│ ❱   6 from maskrcnn_benchmark.modeling.make_layers import conv_with_kaiming_uniform              │
│     7 from maskrcnn_benchmark.layers import DropBlock2D, DyHead                                  │
│     8 from . import fpn as fpn_module                                                            │
│     9 from . import bifpn                                                                        │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/modeling/make_layers.py:10 in <module>             │
│                                                                                                  │
│     7 from torch import nn                                                                       │
│     8 from torch.nn import functional as F                                                       │
│     9 from maskrcnn_benchmark.config import cfg                                                  │
│ ❱  10 from maskrcnn_benchmark.layers import Conv2d, DYReLU                                       │
│    11 from maskrcnn_benchmark.modeling.poolers import Pooler                                     │
│    12                                                                                            │
│    13                                                                                            │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/layers/__init__.py:10 in <module>                  │
│                                                                                                  │
│    7 from .misc import DFConv2d                                                                  │
│    8 from .misc import interpolate                                                               │
│    9 from .misc import Scale                                                                     │
│ ❱ 10 from .nms import nms                                                                        │
│   11 from .nms import ml_nms                                                                     │
│   12 from .nms import soft_nms                                                                   │
│   13 from .roi_align import ROIAlign                                                             │
│                                                                                                  │
│ /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/layers/nms.py:3 in <module>                        │
│                                                                                                  │
│    1 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.                     │
│    2 import torch                                                                                │
│ ❱  3 from maskrcnn_benchmark import _C                                                           │
│    4                                                                                             │
│    5 try:                                                                                        │
│    6 │   import torchvision                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: /home/ec2-user/viper2/GLIP/maskrcnn_benchmark/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: 
_ZN3c104cuda20CUDACachingAllocator9allocatorE
earakely commented 1 year ago

Using the xl model. My jupyter notebook just hangs at the get_code cell.

Loading BLIP...
2023-04-05 23:03:10.085014: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-05 23:03:11.181322: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-04-05 23:03:11.181455: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-04-05 23:03:11.181469: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
This model has some weights that should be kept in higher precision, you need to upgrade `accelerate` to properly deal with them (`pip install --upgrade accelerate`).
Loading checkpoint shards: 100%
2/2 [00:13<00:00, 6.34s/it]
BLIP loaded 
Loading DepthEstimation...
DepthEstimation loaded 
Loading GLIP...
GLIP loaded 
Loading XVLM...
XVLM loaded

and then it hangs on the below cell:

im = load_image('https://wondermamas.com/wp-content/uploads/2020/04/IMG_8950-min-1024x1024.jpg')
query = 'How many muffins can each kid have for it to be fair?'

show_single_image(im)
code = get_code(query)

I don't see any gpu or cpu utilization either:

(base) [ec2-user@ip-172-31-10-144 ~]$ nvidia-smi
Wed Apr  5 23:06:29 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla M60           Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   34C    P0    38W / 150W |   6013MiB /  7680MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla M60           Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   33C    P8    14W / 150W |      3MiB /  7680MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   33C    P0    38W / 150W |   2723MiB /  7680MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   40C    P0    38W / 150W |   2257MiB /  7680MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    162419      C   .../envs/vipergpt/bin/python     6010MiB |
|    2   N/A  N/A    162419      C   .../envs/vipergpt/bin/python     2720MiB |
|    3   N/A  N/A    162419      C   .../envs/vipergpt/bin/python     2254MiB |
+-----------------------------------------------------------------------------+
Tony363 commented 1 year ago

I believe its where your openai accounts computation limit is reached and the script is just constantly sending a request to openai and getting the same response of limit over error

earakely commented 1 year ago

It was because it was sending a request to the codex API which is no longer supported actually.

surisdi commented 1 year ago

Hi,

Yes, unfortunately Codex is no longer supported. Try using GPT3.5 Turbo or GPT4 instead. For that, you will have to change the configuration in this line, and also the prompt file in this line.

@earakely, regarding using Python >=3.9, the setup instructions explicitly create an environment with Python 3.10 (see this line). Did you run the setup instructions as specified in the README? Thanks for the feedback.