Improve documentation concerning the new config files

orbiskcw commented 3 years ago

📚 Documentation Improvements

In short

Concerning: https://detectron2.readthedocs.io/en/latest/tutorials/configs.html Problem: Documentation does not seem to have been updated to reflect the new config files (.py rather than .yaml) Solution: Update the documentation

Problem description

FAIR recently published new Mask R-CNN baselines and this was my first introduction to the new config file that no longer relies on YAML files but on 'raw' .py files. I am trying to load the new baselines using the config files mentioned in the MODEL_ZOO (see this table). For example:

from detectron2 import model_zoo
model = model_zoo.get("new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ.py", trained=True)

This gives

RuntimeError: new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ not available in Model Zoo!

I have installed Detectron2 using the installation instructions. When looking up the documentation on configs, it seems that this has not been updated to reflect the new configs and still solely mentions YAML files.

Proposed solution

It could be that the CONFIG_PATH_TO_URL_SUFFIX dictionary in _ModelZooUrls class still has to be updated and that this is actually a bug (see here), but I find it hard to estimate wheter this is meant behavior (i.e. the new config file should be loaded differently) or a bug due to my limited understanding of the new config files. Either way, I therefore feel like the documentation on readthedocs should be updated to reflect the change from .yaml to .py.

cwognum commented 3 years ago

I think this is actually a bug 🐛

@ppwwyyxx Could you confirm whether this is a bug or expected behavior? I noticed you authored the last commit in the model_zoo.py file and made the changes there for the new .py config files.

VishalBalaji321 commented 3 years ago

This is actually a bug. I had similar issues but when I added the required 'new-baseline' config manually into model_zoo.py file, this was solved.

cwognum commented 3 years ago

I created a pull request to add the mapping from config to model checkpoint for the new baselines. However, I do not think that solves this particular issue. Even though the transition from YAML to PY configs is easy, the documentation could (and should) still be updated to reflect these changes.

orbiskcw commented 3 years ago

Thanks for the help so far!

I indeed do think the documentation should still be updated. I find it especially confusing that the two config types seem to use different keys. For example cfg.MODEL.WEIGHTS vs. cfg.train.init_checkpoint (see model_zoo.py). Could there perhaps be a method to convert between the two formats?

orbiskcw commented 3 years ago

@ppwwyyxx I have been trying to compose a minimal script to understand the usage of the new .py config files in the Detectron2 framework. I see the importance of instantiate(). I expected something like this to work:

cfg = get_config('new_baselines/mask_rcnn_regnety_4gf_dds_FPN_400ep_LSJ.py', True)

model = instantiate(cfg.model)
dataset = instantiate(cfg.dataloader.train)
optimizer = instantiate(cfg.optimizer)

trainer = SimpleTrainer(model, dataset, optimizer)
trainer.train(0, 700000)

I think this is the absolute bare minimum. It does not do any evaluation and does not even save the model. However, this already causes issues that make me suspect that the new config files are not fully supported yet. For example; instantiate(cfg.optimizer) gives TypeError: get_default_optimizer_params() missing 1 required positional argument: 'model'.

There also does not yet seems to be a BaseTrainer sub class that uses the new config format, which generally makes up a quite essential component to a Detectron2 training loop. Is it a correct assumption that the new config files are thus not fully supported yet?

cwognum commented 3 years ago

I share @orbiskcw his intuition. I would be willing to work on this and to create a pull request. Perhaps we can create a new LazyDefaultTrainer class? But I do think we need some pointers and guidance from FAIR on this as it involves some possibly influential changes. @ppwwyyxx Thoughts?

ppwwyyxx commented 3 years ago

https://github.com/facebookresearch/detectron2/blob/master/tools/lazyconfig_train_net.py is the training script to use these configs. We'll add documentation soon.

orbiskcw commented 3 years ago

Awesome! Thank you for the quick response. I completely overlooked that file. Sorry for the inconvenience.

mohsendaoud1 commented 3 years ago

@orbiskcw Can you share your training code please

orbiskcw commented 3 years ago

It's not worth sharing. The provided training script that was just mentioned contains all the information you need. I just updated some configurations so that we can train it on our own data. I added code for the custom dataset and used args.opts for setting some additional parameters. Most importantly:

Set a cfg.train.init_checkpoint (previously cfg.MODEL.WEIGHTS)
Changed SyncBN to just BN because we train with a single GPU
Set cfg.dataloader.train.dataset.names (previously cfg.DATASETS.TRAIN) to the name of our own dataset as registered in the DatasetCatalog. Similarly for cfg.dataloader.test.dataset.names (previously cfg.DATASETS.TEST).

sharing my script will not help you as I barely changed anything and what I did change is application specific

mohsendaoud1 commented 3 years ago

i am new with code thats why i need it

orbiskcw commented 3 years ago

@ppwwyyxx Saving and loading a config file that had a lambda does not seem to work yet. For example:

cfg = get_config('new_baselines/mask_rcnn_regnety_4gf_dds_FPN_400ep_LSJ.py')
LazyConfig.save(cfg, "tmp.yaml")
cfg = LazyConfig.load("tmp.yaml")
model = instantiate(cfg.model)
print(model)

Gives:

yaml.constructor.ConstructorError: while constructing a Python object
cannot find module 'detectron2._cfg_loader8ac1.mask_rcnn_R_50_FPN_100ep_LSJ.py' (No module named 'detectron2._cfg_loader8ac1')
  in "tmp.yaml", line 100, column 18

Manually removing the lambdas from the generated .yaml file does allow it to be loaded and prints the model architecture as expected. However, the model now has no batch normalization. In this case it has to do with this line of code. Although I am not a 100% sure what the best way is to do this, I believe this particular case could be fixed by some changes in get_norm() or FastRCNNConvFCHead.

Generally, however, I am not sure whether and how lambdas could (and should) be supported with the lazy config format. What are your thoughts on this?

ppwwyyxx commented 3 years ago

Lambda should not be saved in a yaml - I don't think there is a reasonable way to do that.

ppwwyyxx commented 3 years ago

We added documentation in https://detectron2.readthedocs.io/en/latest/tutorials/lazyconfigs.html. Feel free to suggest improvements or missing information.

orbiskcw commented 3 years ago

Thank you for this! It reads quite nicely. There is two points I would like to add to the documentation:

What are the best practices for saving and loading config files locally? Take for example the use case in which I want to continue training a provided baseline but want to experiment with some of the configurations and save that updated config. I have seen a total of three options and was wondering how they compare to one another:
- First of all, we can use LazyConfig.load() and LazyConfig.save() to save the 'code' config to a .yaml file.
- Secondly, we could reproduce (part of) the config structure in Python and have our configs stored as code
- Finally, we could use the provided methods in the OmegaConf library (see here)
A little section on how the new config compares to the old config would also be nice. I understand that this is more challenging than I make it sound because there is no 1-to-1 mapping providing an overview like the one used for the yacs config is impossible with this format. However, perhaps some common use cases could still be illustrated to make the transition more smooth. Some examples I had trouble figuring out when I started using the new configs:
- Where to update the batch size?
- Where to update the custom dataset?
- How to change the augmentations?

cwognum commented 3 years ago

I agree with the comments above. An additional question would be whether there are any concrete plans to further expand the support for these new config files throughout the repository? For example, by adding a new DefaultTrainer-like class for lazy configs. If there were to be some sort of overview or roadmap of changes planned / needed, I would happily pick up some of those tasks!

ppwwyyxx commented 3 years ago

What are the best practices for saving and loading config files locally

I agree we should explain more on this topic. Among your solutions: (1) and (3) are actually almost the same. They share the same limitation that the config object can be made unserializable if users choose to use complex objects. I'm not so sure on how (2) can be done.

A little section on how the new config compares to

As you see there isn't a simple 1-1 mapping. Because config follows the code directly, the new config is most suitable for users with enough expertise so that they can directly work with arguments in the code. But still, I think we can list a few examples on the commonly used options.

concrete plans to further expand the support

New features will likely not have support for old config
We'd like to get help on translating more existing yaml's to python, especially those that are sufficiently different from what we have already translated. Translation might uncover new issues.
Some code in lazyconfig_train_net.py can very well be put into some "trainer". So far there is no plan. It is more of a preference but now we feel it's simple enough so that working on this script directly is easier, especially given that the new config intended more for power users.

VishalBalaji321 commented 3 years ago

Hey guys, I am training a simple mask R CNN model for instance segmentation. I found that the new-baselines pre-trained weights offer significantly better accuracy but I am not able to use it as such. I went through the documentation of @ppwwyyxx and lazyconfig_train_net.py but as a beginner, I couldn't understand much. It would be really helpful if you guys could provide me with a simple script how to initiate the entire config file including parameters max_iters, steps and gamma from scratch and how to start training the model.

Cheers, Vishal Balaji

orbiskcw commented 3 years ago

Hey @VishalBalaji321, it seems like the lazyconfig_train_net.py CLI can do exactly what you are looking for. I am not sure what information you are missing. Please note that using the default argument parser used in the lazyconfig_train_net.py script gives you the opts parameter which is then used to specify the overrides. Meaning, that you can simply override the config using the path.key=value syntax. For example:

# Overrides the maximum number of iterations by 100 and the learning rate by 0.1
python lazyconfig_train_net.py --config-file ... train.max_iter=100 optimizer.lr=0.1

Of course, you coud also do this in code. Similarly to what is done here.

# Overrides the maximum number of iterations by 100 and the learning rate by 0.1
cfg.train.max_iter = 100
cfg.optimizer.lr = 0.1

Without any concrete questions, I can't really help you more than that. I'm afraid that, like @ppwwyyxx said as well, this new config requires you to understand the code a bit better than before. But I have been working with it for a little while now and I actually have been enjoying it more and more. It's quite powerful!

VishalBalaji321 commented 3 years ago

It's not worth sharing. The provided training script that was just mentioned contains all the information you need. I just updated some configurations so that we can train it on our own data. I added code for the custom dataset and used args.opts for setting some additional parameters. Most importantly:

Set a cfg.train.init_checkpoint (previously cfg.MODEL.WEIGHTS)

Changed SyncBN to just BN because we train with a single GPU

Set cfg.dataloader.train.dataset.names (previously cfg.DATASETS.TRAIN) to the name of our own dataset as registered in the DatasetCatalog. Similarly for cfg.dataloader.test.dataset.names (previously cfg.DATASETS.TEST).

sharing my script will not help you as I barely changed anything and what I did change is application specific

Thanks a lot for the previous reply @orbiskcw , it did help me a lot. Now I am facing issues with 'SyncBN' normalization. Could you pls explain how to change it to just 'BN' for single GPU? (For Info: I am planning on using newbaselines/R101_FPN_400ep model

orbiskcw commented 3 years ago

Happy to have helped. I am pretty sure that the change depends on the model you are using. For me, it was

# Using the python syntax here, but as I have shown earlier 
# it looks similar if you provide these overrides using the CLI
cfg.model.backbone.bottom_up.norm = "BN" 
cfg.model.backbone.norm = "BN"

To help you find the right keys, I would recommend doing one of two things:

Either clone the Detectron2 repository and using your favorite IDE go through the hierarchical definition of the .py configs manually to search for the configuration you are interested in.
Or load the config using LazyConfig.load(args.config_file) and immediately save it to a .yaml file with LazyConfig.save('/local/path/to/config.yaml). This makes it a bit easier because everything is one place.

Note that because the lazy configs closely follow the code, not all configurations you are looking for will be present in the configs or easy to find with either of these two options. I'm afraid there is no other solution than to just bite the bullet and dig in the Detectron2 code! I think you will find it is nicely structured and quite easy to comprehend once you get going! :wink:

orbiskcw commented 3 years ago

@VishalBalaji321 In case you have not seen this yet, the documentation was updated yesterday: https://detectron2.readthedocs.io/en/latest/tutorials/lazyconfigs.html It explains a lot of the things I tried to explain to you as well, but better! :wink:

@ppwwyyxx Great work by you and your colleagues. It feels a lot more complete now. Nothing comes to mind anymore that could be added.

orbiskcw commented 3 years ago

@ppwwyyxx One more question that recently came up while working with the detectron2 repo: What is the best way to extend a config from the model zoo? I expected something like this to work:

# In /path/to/custom_config.py

from detectron2.model_zoo.configs.new_baselines.mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ import  (
    dataloader,
    lr_multiplier,
    model,
    optimizer,
    train,
)

train.max_iter = 10

But if I then try to load the new config with:

LazyConfig.load('/path/to/custom_config.py')

I get:

  File "/home/cas/Documents/repositories/detectron2-model-training/detectron2_model_training/config.py", line 98, in load
    return LazyConfig.load(config_path)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 202, in load
    exec(compile(content, filename, "exec"), module_namespace)
  File "configs/default.py", line 1, in <module>
    from detectron2.model_zoo.configs.new_baselines.mask_rcnn_regnetx_4gf_dds_FPN_100ep_LSJ import (  # noqa
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 145, in new_import
    return old_import(name, globals, locals, fromlist=fromlist, level=level)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/model_zoo/configs/new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_100ep_LSJ.py", line 1, in <module>
    from .mask_rcnn_R_50_FPN_100ep_LSJ import (
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/config/lazy.py", line 145, in new_import
    return old_import(name, globals, locals, fromlist=fromlist, level=level)
  File "/home/cas/Documents/repositories/detectron2-model-training/.venv/lib/python3.8/site-packages/detectron2/model_zoo/configs/new_baselines/mask_rcnn_R_50_FPN_100ep_LSJ.py", line 13, in <module>
    train.init_checkpoint = ""
AttributeError: 'dict' object has no attribute 'init_checkpoint'

Could you confirm whether this is expected behavior or a bug? Could you clarify if it is possible to do this and if so, what the conditions / requirements are?

orbiskcw commented 3 years ago

Something like this does work, but feels very cumbersome:

config = get_config('new_baselines/mask_rcnn_regnetx_4gf_dds_FPN_400ep_LSJ.py')

train = config.train
optimizer = config.optimizer
model = config.model
dataloader = config.dataloader
lr_multiplier = config.lr_multiplier
del config

train.max_iter = 10

orbiskcw commented 3 years ago

@ppwwyyxx Do you see the chance to still respond to the above question?

ppwwyyxx commented 3 years ago

The failure is currently expected because when doing LazyConfig.load we only turn relative imports into config loading.

However it would be nice to enable from detectron2.model_zoo.configs.xxx import .... This should be possible by modifying our custom import function at https://github.com/facebookresearch/detectron2/blob/fe4d35b0a8b6b37d45162a68f82e73f4bf7b1ba1/detectron2/config/lazy.py#L124-L130 which currently only handles relative import.

orbiskcw commented 3 years ago

Clear! I have taken a look at the method you referenced, but this is some Python wizardry I am not really familiar with and I wouldn't know where to start to support absolute imports. I might take a closer look at it once I have a bit more time, but for now I will stick to the workaround. Thank you for your reply! :+1:

FeriBolour commented 2 years ago

Hello Everyone,

Thank you for all the info mentioned in this issue. It really helped me in understanding how to use LazyConfig to use the architectures in the new_baselines.

However I am still having an issue which I don't know if it is due to my lack of understanding of the new configs.

I am using the _lazyconfig_trainnet.py for my own custom coco format dataset. I've already used this dataset with the DefaultTrainer class and it works perfectly. However I am having an issue with this dataset when it comes to this script.

For now, I haven't changed anything in _lazyconfig_trainnet.py except the main function. This is how the main function looks like now:

def main(args):
    register_coco_instances("CottonImaging_train", {}, '/home/avl/Projects/Cotton Imaging Project/Data/Datasets02272022/7030_images/TrainingSet2_imbalanced_70.json', "/home/avl/Projects/Cotton Imaging Project/Data/Datasets02272022/7030_images/training_images")

    cfg = get_config("new_baselines/mask_rcnn_R_101_FPN_400ep_LSJ.py")
    cfg.model.backbone.bottom_up.stages = detectron2.modeling.ResNet.make_default_stages(depth=101, norm='BN', stride_in_1x1=True)
    cfg.model.backbone.norm = "BN"
    cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')
    cfg.train.max_iter=10000
    cfg.train.output_dir='./MaskRCNN ResNet101 400ep LSJ'
    cfg.model.roi_heads.num_classes = 4
    cfg.optimizer.lr=0.001
    cfg.dataloader.train.total_batch_size = 12
    default_setup(args, cfg)    
    do_train(cfg)

When I run this script, this is the error I get:

[03/31 17:04:17 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in training: [ResizeScale(min_scale=0.1, max_scale=2.0, target_height=1024, target_width=1024), FixedSizeCrop(crop_size=[1024, 1024]), RandomFlip()]
[03/31 17:04:18 fvcore.common.checkpoint]: No checkpoint found. Initializing model from scratch
[03/31 17:04:18 d2.engine.train_loop]: Starting training from iteration 0
ERROR [03/31 17:04:19 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 149, in train
    self.run_step()
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 391, in run_step
    data = next(self._data_loader_iter)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/common.py", line 179, in __iter__
    for d in self.dataset:
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/avl/.local/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
omegaconf.errors.ConfigAttributeError: Caught ConfigAttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/avl/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/common.py", line 43, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/utils/serialize.py", line 23, in __call__
    return self._obj(*args, **kwargs)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/dataset_mapper.py", line 168, in __call__
    annos = [
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/dataset_mapper.py", line 169, in <listcomp>
    utils.transform_instance_annotations(
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/data/detection_utils.py", line 284, in transform_instance_annotations
    bbox = BoxMode.convert(annotation["bbox"], annotation["bbox_mode"], BoxMode.XYXY_ABS)
  File "/home/avl/.local/lib/python3.8/site-packages/detectron2/structures/boxes.py", line 80, in convert
    arr = box.clone()
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/listconfig.py", line 176, in __getattr__
    self._format_and_raise(
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/home/avl/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: ListConfig does not support attribute access
    full_key: [130].annotations[0].bbox[clone]
    object_type=list

I register my dataset the same way and use it for DeafultTrainer and I don't get any errors.

Here's how my annotations look like for reference if needed: I'd appreciate any kind of help with this. Please let me know if I need to provide more info.

ppwwyyxx commented 2 years ago

cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')

The value of this config should be a config object, but not the actual data itself. So you'd want

    from detectron2.config import LazyConfig as L
    cfg.dataloader.train.dataset = L(detectron2.data.get_detection_dataset_dicts)(names='CottonImaging_train')

See examples in https://github.com/facebookresearch/detectron2/blob/6886f85baee349556749680ae8c85cdba1782d8e/configs/common/data/coco.py#L15-L16

FeriBolour commented 2 years ago

cfg.dataloader.train.dataset= detectron2.data.get_detection_dataset_dicts(names='CottonImaging_train')
The value of this config should be a config object, but not the actual data itself. So you'd want
    from detectron2.config import LazyConfig as L
    cfg.dataloader.train.dataset = L(detectron2.data.get_detection_dataset_dicts)(names='CottonImaging_train')
See examples in

https://github.com/facebookresearch/detectron2/blob/6886f85baee349556749680ae8c85cdba1782d8e/configs/common/data/coco.py#L15-L16

Thank You for the very quick reply. It solved my problem and I was able to successfully train my model.

Now a quick question that might be easy. I just want to know if there are any specific functions that I can use.

How can I predict on single images with my saved model? Almost anywhere I look, they use the DefaultPredictor class which I don't think we can use with these config files. (If we can, I'd appreciate if you tell me how). Are there any other classes like the DefaultPredictor that I could use for this purpose. If not, how would you suggest that I do the predictions.

Thank You

EDIT: I tried to solve this issue by using the following script:

cfg = get_config("new_baselines/mask_rcnn_R_50_FPN_400ep_LSJ.py")
cfg.model.backbone.bottom_up.stages = detectron2.modeling.ResNet.make_default_stages(depth=50, norm='BN', stride_in_1x1=True)
cfg.model.backbone.norm = "BN"
cfg.model.backbone.bottom_up.stem = detectron2.modeling.backbone.BasicStem(in_channels=3, norm='BN', out_channels=64)
cfg.model.roi_heads.num_classes = 4
cfg.train.init_checkpoint = './model_final.pth'
model = instantiate(cfg.model)
model.to(cfg.train.device)
model = create_ddp_model(model)
DetectionCheckpointer(model).load(cfg.train.init_checkpoint)

model.eval()

def mapper(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)
    dicts = []
    for i in range(len(dataset_dict)):
        image = utils.read_image(dataset_dict[i]["file_name"], format="BGR")
        temp = image.copy()
        utils.check_image_size(dataset_dict[i], temp)
        image = torch.from_numpy(temp)
        dicts.append({
           "image": image,
           'height': 720,
           'width': 1280,
        })
    return dicts

dataset_dict = detectron2.data.get_detection_dataset_dicts('CottonImaging_test')
data= mapper(dataset_dict)

with torch.no_grad():
  outputs = model(data)

But I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_3575/3931202212.py in <module>
      3 
      4 with torch.no_grad():
----> 5   outputs = model(data)

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in forward(self, batched_inputs)
    144         """
    145         if not self.training:
--> 146             return self.inference(batched_inputs)
    147 
    148         images = self.preprocess_image(batched_inputs)

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in inference(self, batched_inputs, detected_instances, do_postprocess)
    197         assert not self.training
    198 
--> 199         images = self.preprocess_image(batched_inputs)
    200         features = self.backbone(images.tensor)
    201 

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in preprocess_image(self, batched_inputs)
    223         """
    224         images = [x["image"].to(self.device) for x in batched_inputs]
--> 225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)
    227         return images

~/.local/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py in <listcomp>(.0)
    223         """
    224         images = [x["image"].to(self.device) for x in batched_inputs]
--> 225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)
    227         return images

RuntimeError: The size of tensor a (720) must match the size of tensor b (3) at non-singleton dimension 0

I don't know if this is the right way to do the predictions. But if it is, I don't exactly understand how to resolve this error.

ppwwyyxx commented 2 years ago

As documented in https://detectron2.readthedocs.io/en/latest/tutorials/models.html#model-input-format the image of model's input has to be in (C, H, W) format while in your code it's (H, W, C).

DefaultPredictor would need a patch like https://github.com/facebookresearch/detectron2/pull/3755 to work with new configs.

FrancescoFarinola commented 2 years ago

If anyone is looking for a solution for doing inference on single images remember to:

Convert the image to your model.input_format (usually BGR since openCV converts images as default in BGR) im = cv2.imread("path/to/image.jpg")
Transform the BGR image into a tensor with image = torch.from_numpy(im)
Permute the tensor as the model input format is (C, H, W) so: image_tensor = image.permute(2,0,1)
Do inference - remember that when doing inference on custom models you need to pass a list of dicts, so if you are doing inference on a single image:
```
model.eval()
with torch.no_grad():
    outputs = model(d)
```
where d is d = [{'image' : image_tensor}]

Bill4869 commented 2 years ago

Hello. Sorry for bothering. I have looked at the documentation but I still could not test pretrained new baselines model on one image. I was wondering if anyone could tell me how to do it?

Thank you in advances.

bouachalazhar commented 1 year ago

Hello. Sorry for bothering. I have looked at the documentation but I still could not test pretrained new baselines model on one image. I was wondering if anyone could tell me how to do it?

Thank you in advances.

Same problem

rezat96 commented 1 year ago

Hi, I have carefully read the docs and the tutorial, but is there an equivalent of cfg.INPUT.MASK_FORMAT = 'bitmask' for the new baselines? I could use it with previous pipelines but with the new one I cant train on my custom dataset @ppwwyyxx

yanghoonkim commented 11 months ago

@rezat96 cfg.dataloader.train.mapper['instance_mask_format'] = 'bitmask'

CA4GitHub commented 10 months ago

As documented in https://detectron2.readthedocs.io/en/latest/tutorials/models.html#model-input-format the image of model's input has to be in (C, H, W) format while in your code it's (H, W, C).

DefaultPredictor would need a patch like #3755 to work with new configs.

Is there a way to convert new configs to old configs that will work with DefaultPredictor?

facebookresearch / detectron2