search

facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.
https://detectron2.readthedocs.io/en/latest/
Apache License 2.0
30.13k stars 7.42k forks source link

Example for rotated faster rcnn #21

Open cjt222 opened 4 years ago

cjt222 commented 4 years ago

Thanks for great work, i have a question, how can i set config file to detect oriention target in faster rcnn or other network?

ppwwyyxx commented 4 years ago

What do you mean by "oriention target"?

If you mean objects that are often not horizontal/vertically aligned, like texts, detectron2 supports rotated bounding boxes. But an example is not available now.

cjt222 commented 4 years ago

Thanks for your reply!! I still have some question: 1、what kind of network can support rotated bounding boxes？ 2、what kind of datasets have you tried when detecting with rotated bounding boxes and how to calculate mAP while eval and could you give me a baseline? 3、Could you share papers in rotated bounding boxes in detectron2?

ppwwyyxx commented 4 years ago

There will be more details when a complete training example is released.

Aswantm commented 4 years ago

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. The code will be released.

issa-tingzon commented 4 years ago

Hi, would like to follow up on this. Would you have a minimal working example on training and evaluating models using rotated bounding boxes? Thanks!

freesunshine commented 4 years ago

We almost cry out for it! A minimal working example is needed!Thanks!

WeihongM commented 4 years ago

@ppwwyyxx Any updates?

edwardchaos commented 4 years ago

I'm attempting to do this as well, so far I've found these configs:

cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN" cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"

Currently, the angles in anchors are at 90 degree intervals so i'll need to add more angles.

I'm using the faster_rcnn_R_101_FPN_3x pretrained network.

The issue now is using detectron2's dataloader with rotated bounding box annotations. In the documentation it says that box must be a member of detectron2.structures.BoxMode which only supports horizontal bounding box annotations.

To get it to work now I'm thinking i'll have to use another way of loading data. @ppwwyyxx do you think it will be available in days, weeks, or months?

ppwwyyxx commented 4 years ago

Yes a custom dataloader is needed to provide rotated boxes as input. We don't have a concrete timeline.

edwardchaos commented 4 years ago

Update:

I found another config that needs to be overridden: MODEL.RPN.BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0, 1.0)

The default configs file doesn't have a section for RRPN, assuming it's intentional and we're supposed to override RPN configs

freesunshine commented 4 years ago

@edwardchaos Appreciate you for working on it! Can you keep on updating your progress here? That will be very helpful!

edwardchaos commented 4 years ago

Yup, now I'm taking a step back and getting horizontal bounding boxes to work first, found another config that needs to be changed for rotated boxes:

MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"

edwardchaos commented 4 years ago

ok I got rotated bounding box working on DOTA dataset: https://captain-whu.github.io/DOTA/dataset.html

for training, after several epochs, it'll fail but I'm pretty sure the issue is with respect to invalid ground truth bounding boxes in my dataset.

@ppwwyyxx do you suggest I first train rrpn and then rroi heads rather than end to end?

For inference, getdeltas() calls torch.div() which resulted in a pytorch error. I got past this by changing the in place version to torch.div(). Not sure if this is considered a bug.

I wrote a custom dataloader and data augmentation using imgaug. Dataloader prepares the data in detectron2 input format (list of dictionaries)

As far as this issue goes, here are my configs that overwrite the base rcnn fpn config:

BASE: "./Base-RCNN-FPN.yaml" MODEL: PIXEL_MEAN: [103.530, 116.280, 123.675] # Pixel mean of imagenet dataset R/G/B? WEIGHTS: "./model_final_a3ec72.pkl" MASK_ON: False RESNETS: DEPTH: 101 PROPOSAL_GENERATOR: NAME: "RRPN" RPN: HEAD_NAME: "StandardRPNHead" BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0, 1.0) ANCHOR_GENERATOR: NAME: "RotatedAnchorGenerator" ANGLES: [[0, 30, 60, 90]] ASPECT_RATIOS: [[0.5, 1.0, 2.0]] ROI_HEADS: NAME: "RROIHeads" NUM_CLASSES: 15 # number of foreground classes ROI_BOX_HEAD: POOLER_TYPE: "ROIAlignRotated" BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0, 10.0) SOLVER: STEPS: (210000, 250000) MAX_ITER: 10 # number of epochs ( I modified the plain net train from iterations to epochs) CHECKPOINT_PERIOD: 1 # number of epochs

WeihongM commented 4 years ago

@edwardchaos Hello, how do you prepare your data loader, can you share the script here? Also, how you prepare the angle theta. Thanks.

MaximKuklin commented 4 years ago

Hi @edwardchaos, many thanks for your config! I faced the same problem with torch.div_() func. So, this can be fixed by putting torch.no_grad(). This helps because we are working with self.weights field, which contains gradient.

with torch.no_grad():
    wx, wy, ww, wh, wa = self.weights
    dx, dy, dw, dh, da = torch.unbind(deltas, dim=1)

    dx.div_(wx)
    dy.div_(wy)
    dw.div_(ww)
    dh.div_(wh)
    da.div_(wa)

I found solution here: https://discuss.pytorch.org/t/runtimeerror-output-nr-0-assert-failed-at-pytorch-torch-csrc-autograd-variable-cpp-196-please-report-a-bug-to-pytorch/39604/2

Moreover, can you share the info about how long does it take to train your model and how many images in dataset do you have, please?

edwardchaos commented 4 years ago

@WeihongM , this should help: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html I'm not sure I can share the code but basically you'll want to get a list of dictionaries where each dictionary corresponds to a training sample with keys "image", "gt_instance", etc. Image should bea pytorch tensor and gt_instances is an instances object. Part of debugging, you can step through the network and check the exact keys it expects. The design pattern makes stepping through the code a little confusing at first as it jumps from place to place.

Angle representation is described in the comment here. When calculating angle, you'll come across atan or atan2. Should use atan2.

@MaximKuklin , I've trained it on DOTA 1.0 train set splitting the image down to 256x256 squares and removing any samples that don't have anything objects in them. Hardware is 2080Ti and 8 core cpu. Takes about a day or 2 for ~200k iterations (minibatch size 1). I'd train it for a week total; it depends on your dataset, and learning configurations.

I've also adapted this rotated faster rcnn to SCRDet and found it to perform better.

janzd commented 4 years ago

Are rotated bounding boxes available for Mask R-CNN too?

zhenglilei commented 4 years ago

Looking forward to the example in 2019. 😄

Kongsea commented 4 years ago

Look forward to it too.

kakaluote commented 4 years ago

hello @edwardchaos i use your config, and write my dataloader, i can start to train r2cnn, but the train crashed after several step, CUDA out of memory

[02/22 19:08:27 d2.utils.events]: eta: 0:53:32 iter: 16 total_loss: 2.584 loss_cls: 0.722 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.209 time: 0.4244 data_time: 0.0099 lr: 0.000017 max_mem: 3168M [02/22 19:08:27 d2.utils.events]: eta: 0:54:36 iter: 17 total_loss: 2.656 loss_cls: 0.709 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.210 time: 0.4257 data_time: 0.0094 lr: 0.000018 max_mem: 3168M [02/22 19:08:28 d2.utils.events]: eta: 0:53:31 iter: 18 total_loss: 2.584 loss_cls: 0.695 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.211 time: 0.4230 data_time: 0.0090 lr: 0.000019 max_mem: 3168M ERROR [02/22 19:08:28 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/home/1T/workspace/detectron2/tools/../detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/home/1T/workspace/detectron2/tools/../detectron2/engine/train_loop.py", line 214, in run_step loss_dict = self.model(data) File "/home/kakaluote/anaconda3/envs/pytorch-py3-lastest/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, *kwargs) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/meta_arch/rcnn.py", line 123, in forward proposals, proposal_losses = self.proposal_generator(images, features, gt_instances) File "/home/kakaluote/anaconda3/envs/pytorch-py3-lastest/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(input, **kwargs) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward losses = outputs.losses() File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rpn_outputs.py", line 322, in losses gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth() File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn_outputs.py", line 219, in _get_ground_truth matched_idxs, gt_objectness_logits_i = self.anchor_matcher(match_quality_matrix) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 95, in call self.set_low_qualitymatches(match_labels, match_quality_matrix) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 115, in set_low_qualitymatches match_quality_matrix == highest_quality_foreach_gt[:, None] RuntimeError: CUDA out of memory. Tried to allocate 2.53 GiB (GPU 0; 7.76 GiB total capacity; 2.08 GiB already allocated; 1.21 GiB free; 1.42 GiB cached) [02/22 19:08:28 d2.engine.hooks]: Overall training speed: 17 iterations in 0:00:07 (0.4509 s / it) [02/22 19:08:28 d2.engine.hooks]: Total training time: 0:00:08 (0:00:00 on hooks)

Do you have any suggestions？need your help， thanks

edwardchaos commented 4 years ago

Try reducing your batch size or input volume or network

kakaluote commented 4 years ago

yes，batchsize is 1， input is （600，） ，backbone is res50fpn same crash。。。 Could it be a crash caused by input data? i'm sure the angle is (-180,180)

minhajul-arifin-badhon commented 4 years ago

Hello @ppwwyyxx . I am using these configs:

`cfg = get_cfg() cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml")) cfg.DATASETS.TRAIN = ("train",) cfg.DATASETS.TEST = ("val",)

cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo

cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"

cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead" cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (1.0, 1.0, 1.0, 1.0, 1.0)

cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator" cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[0, 30, 60, 90]] cfg.MODEL.ANCHOR_GENERATOR.ASPECT_RATIOS = [[0.5, 1.0, 2.0]]

cfg.MODEL.ROI_HEADS.NAME = "RROIHeads" cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128 # faster, and good enough for this toy dataset (default: 512) cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class

cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated" cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)

cfg.SOLVER.MAX_ITER = 300 # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset`

but getting the following errors after running:

[02/24 17:55:30 d2.engine.train_loop]: Starting training from iteration 0 ERROR [02/24 17:55:33 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 214, in run_step loss_dict = self.model(data) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/meta_arch/rcnn.py", line 123, in forward proposals, proposal_losses = self.proposal_generator(images, features, gt_instances) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward losses = outputs.losses() File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rpn_outputs.py", line 322, in losses gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth() File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rrpn_outputs.py", line 234, in _get_ground_truth anchors_i.tensor, matched_gt_boxes.tensor File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/box_regression.py", line 149, in get_deltas target_boxes, dim=1 ValueError: not enough values to unpack (expected 5, got 4) [02/24 17:55:33 d2.engine.hooks]: Total training time: 0:00:02 (0:00:00 on hooks)

I am not sure how to fix this. I assume I don't need a custom dataloader anymore. (not using) Can you please help? Thanks.

kakaluote commented 4 years ago

update I found that the number of objects in image is related to the memory usage on my dataset，some image have 47 objects, and the training always crash at these images. 27 objects will use 4171M GPU memory. more info: crash place is same as below

File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn_outputs.py", line 219, in get_ground_truth matched_idxs, gt_objectness_logits_i = self.anchor_matcher(match_quality_matrix) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 95, in call self.set_low_quality_matches(match_labels, match_quality_matrix) File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 115, in set_low_quality_matches match_quality_matrix == highest_quality_foreach_gt[:, None] RuntimeError: CUDA out of memory. Tried to allocate 2.53 GiB (GPU 0; 7.76 GiB total capacity; 2.08 GiB already allocated; 1.21 GiB free; 1.42 GiB cached)

@ppwwyyxx Why the number of targets is related to memory？

cbasavaraj commented 4 years ago

I am yet to try rotated bounding boxes, but hoping to soon. Good question by @kakaluote. In the normal faster-rcnn, the training batch sizes are fixed both for the RPN and the second stage (RCNN). So even if we have more GT boxes, and hence more foreground proposals / ROIs, there should be no proportional increase in memory allocated. At least that's what I would think.

st7ma784 commented 4 years ago

@minhajul-arifin-badhon It looks like when the model is beginning training your dataset only has 4 values where it should have 5. From what I can tell, RRCNN methods require an extra BBox value for rotation (I suspect -im about to look and play around- that its x,y,w,h, angle) BoxMode.XYWHA_ABS is what you want?

minhajul-arifin-badhon commented 4 years ago 
    record = {}
    record["file_name"] = filename
    record["image_id"] = idx
    record["height"] = 1024
    record["width"] = 1024

    objs = []
    for row_index, row in df_group.iterrows():
        obj = {
            "bbox": [row['x_c'], row['y_c'], row['w'], row['h'], row['a']],
            "bbox_mode": BoxMode.XYWHA_ABS,
            "category_id": 0,
            "iscrowd": 0
        }
        objs.append(obj)

    record["annotations"] = objs

@st7ma784 yes, I want to train with rotated bounding boxes. I have used the above code to get each of the annotations. I think I am using angles properly. I also tried to train with only two images and checked that the data I am providing is correct. Not sure what I am missing. Have you tried training with rotated bounding boxes?

st7ma784 commented 4 years ago

I havent got it working yet(being optimistic here :-P ). my guess is that image has something to do with it

Amir22010 commented 4 years ago

help me to resolve this issue,,,i am using BoxMode.XYWHA_ABS ERROR [03/01 19:44:57 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/content/detectron2_repo/detectron2/engine/train_loop.py", line 132, in train self.run_step() File "/content/detectron2_repo/detectron2/engine/train_loop.py", line 215, in run_step loss_dict = self.model(data) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, *kwargs) File "/content/detectron2_repo/detectron2/modeling/meta_arch/rcnn.py", line 124, in forward proposals, proposal_losses = self.proposal_generator(images, features, gt_instances) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, **kwargs) File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward losses = outputs.losses() File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rpn_outputs.py", line 327, in losses gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth() File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rrpn_outputs.py", line 238, in _get_ground_truth anchors_i.tensor, matched_gt_boxes.tensor File "/content/detectron2_repo/detectron2/modeling/box_regression.py", line 157, in get_deltas target_boxes, dim=1 ValueError: not enough values to unpack (expected 5, got 4) [03/01 19:44:57 d2.engine.hooks]: Total training time: 0:00:00 (0:00:00 on hooks)

ValueError Traceback (most recent call last)

in () 132 trainer = DefaultTrainer(cfg) 133 trainer.resume_or_load(resume=False) --> 134 trainer.train() 156 target_ctr_x, target_ctr_y, target_widths, target_heights, target_angles = torch.unbind( --> 157 target_boxes, dim=1 158 ) 159
st7ma784 commented 4 years ago

so, interesting development. print(DatasetCatalog.list()) DatasetCatalog.register("Train", returnTRAINRECORDS) MetadataCatalog.get("Train").set(thing_classes=list(ClassesNames.keys()))

try: convert_to_coco_json("Train", ''.join("./output/"+"Train"+"_coco_format.json")) print("Succeeded in making Train_coco_format.json") MetadataCatalog.get("Train").set(json_file=''.join("./output/"+"Train"+"_coco_format.json")) except: pass`

Is what Im using to load my dataset into a coco format JSON file, meaning it can be used later in an evaluator. I'm suspicious that this doesnt get to the print in the "try" section when the Bbox mode is XYWHA_ABS ... might this be the cause of only getting 4 values when resolving deltas?

Amir22010 commented 4 years ago

with my own code to generate data dict

i am getting dataset_dicts

[{'annotations': [{'bbox': [465.1667, 696.8333, 757.0, 67.0, 0.0], 'bbox_mode': <BoxMode.XYWHA_ABS: 4>, 'category_id': 1, 'iscrowd': 0}, {'bbox': [238.4438, 263.116, 343.7455, 38.4476, 0.02], 'bbox_mode': <BoxMode.XYWHA_ABS: 4>, 'category_id': 0, 'iscrowd': 0}], 'file_name': '/content/gdrive/My Drive/app/detectro/images/ad7ab3c1-1e87-43e2-af06-c53ed76e9e82.jpg', 'height': 1010, 'image_id': 0, 'width': 898}, {'annotations': [{'bbox': [425.295, 612.7028, 247.3418, 25.9827, 3.021593], 'bbox_mode': <BoxMode.XYWHA_ABS: 4>, 'category_id': 1, 'iscrowd': 0}, {'bbox': [82.0213, 480.0, 115.0, 26.0, 2.991593], 'bbox_mode': <BoxMode.XYWHA_ABS: 4>, 'category_id': 0, 'iscrowd': 0}], 'file_name': '/content/gdrive/My Drive/app/detectro/images/fa981233-a7f2-4bb5-84b1-8f7a5cc06011.jpg', 'height': 812, 'image_id': 1, 'width': 575},

then i register with below code is it correct??????

from detectron2.data import DatasetCatalog, MetadataCatalog for d in ["train"]: DatasetCatalog.register("notic_" + d, lambda d=d: get_data_dicts(imagepaths)) MetadataCatalog.get("notic" + d).set(thing_classes=classes)

breznak commented 4 years ago

Guys, I've been working on broken RRPN and only now found this useful thread! If you can help me review and implement needed missing piece in the PR #996 , I think we can finally make this work out of the box :card_file_box:

ryangooch commented 4 years ago

@breznak Thanks for your work on this! I've been trying to get this workflow working too.

I'm not sure if this is the right place to discuss (or if it should be on your PR) but I have been running into a problem with your code. The error occurs when I am trying to train (I'm using some of the above config options as well as my own data loaded):

<ipython-input-13-381075003630> in <module>
      1 os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
----> 2 trainer = DefaultTrainer(cfg)
      3 trainer.resume_or_load(resume=False)

c:\users\rgooch\documents\detectron2_repo\detectron2\engine\defaults.py in __init__(self, cfg)
    249             setup_logger()
    250         # Assume these objects must be constructed in this order.
--> 251         model = self.build_model(cfg)
    252         optimizer = self.build_optimizer(cfg, model)
    253         data_loader = self.build_train_loader(cfg)

c:\users\rgooch\documents\detectron2_repo\detectron2\engine\defaults.py in build_model(cls, cfg)
    395         Overwrite it if you'd like a different model.
    396         """
--> 397         model = build_model(cfg)
    398         logger = logging.getLogger(__name__)
    399         logger.info("Model:\n{}".format(model))

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\meta_arch\build.py in build_model(cfg)
     17     """
     18     meta_arch = cfg.MODEL.META_ARCHITECTURE
---> 19     return META_ARCH_REGISTRY.get(meta_arch)(cfg)

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\meta_arch\rcnn.py in __init__(self, cfg)
     32         self.device = torch.device(cfg.MODEL.DEVICE)
     33         self.backbone = build_backbone(cfg)
---> 34         self.proposal_generator = build_proposal_generator(cfg, self.backbone.output_shape())
     35         self.roi_heads = build_roi_heads(cfg, self.backbone.output_shape())
     36         self.vis_period = cfg.VIS_PERIOD

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generator\build.py in build_proposal_generator(cfg, input_shape)
     22         return None
     23 
---> 24     return PROPOSAL_GENERATOR_REGISTRY.get(name)(cfg, input_shape)

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generator\rrpn.py in __init__(self, cfg, input_shape)
     26 
     27     def __init__(self, cfg, input_shape: Dict[str, ShapeSpec]):
---> 28         super(RRPN, self).__init__(cfg, input_shape)
     29         # three lines below added for pull request 996
     30         assert(isinstance(self.anchor_generator, RotatedAnchorGenerator)), \

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generator\rpn.py in __init__(self, cfg, input_shape)
    119             cfg, [input_shape[f] for f in self.in_features]
    120         )
--> 121         self.box2box_transform = Box2BoxTransform(weights=cfg.MODEL.RPN.BBOX_REG_WEIGHTS)
    122         self.anchor_matcher = Matcher(
    123             cfg.MODEL.RPN.IOU_THRESHOLDS, cfg.MODEL.RPN.IOU_LABELS, allow_low_quality_matches=True

c:\users\rgooch\documents\detectron2_repo\detectron2\modeling\box_regression.py in __init__(self, weights, scale_clamp)
     34         """
     35         self.weights = weights
---> 36         assert(len(weights)==4),"Expecting a 4-elements tuple (for scaling (dx,dy,dw,dh))"
     37         self.scale_clamp = scale_clamp
     38 

AssertionError: Expecting a 4-elements tuple (for scaling (dx,dy,dw,dh))

I suspect that the RRPN class in rrpn.py is trying to initialize the RPN class, which has in its __init __ block a call to Box2BoxTransform, which expects 4 weights, and receives the 5 weights from RRPN.

What do you think?

ppwwyyxx commented 4 years ago

Putting out a full example is not on our todo list for near term, but to unblock some users, what you need to train a Rotated Faster R-CNN is the following changes to config:

MODEL:
  ANCHOR_GENERATOR:
    NAME: RotatedAnchorGenerator
    ANGLES: [[-90,-60,-30,0,30,60,90]]
  PROPOSAL_GENERATOR:
    NAME: RRPN
  RPN:
    BBOX_REG_WEIGHTS: (1,1,1,1,1)
  ROI_BOX_HEAD:
    POOLER_TYPE: ROIAlignRotated
    BBOX_REG_WEIGHTS: (10,10,5,5,1)
  ROI_HEADS:
    NAME: RROIHeads

plus a custom mapper in the data loader to handle rotated boxes. Some notable changes needed in the mapper:

Plus a custom dataset that contains boxes with mode XYWHA.

st7ma784 commented 4 years ago

Awesome! Many thanks, as for an example, I'm working on a dissertation in colab notebook environment, when finished I will happily share a cleaned up version here as an example.

Get Outlook for Androidhttps://aka.ms/ghei36

From: Yuxin Wu notifications@github.com Sent: Friday, March 6, 2020 12:58:27 AM To: facebookresearch/detectron2 detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.uk; Mention mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Putting out a full example is not on our todo list for near term, but to unblock some users, what you need to train a Rotated Faster R-CNN is the following changes to config:

MODEL: ANCHOR_GENERATOR: NAME: RotatedAnchorGenerator ANGLES: [[-90,-60,-30,0,30,60,90]] PROPOSAL_GENERATOR: NAME: RRPN RPN: BBOX_REG_WEIGHTS: (1,1,1,1,1) ROI_BOX_HEAD: POOLER_TYPE: ROIAlignRotated BBOX_REG_WEIGHTS: (10,10,5,5,1) ROI_HEADS: NAME: RROIHeads

plus a custom mapper in the data loaderhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdetectron2.readthedocs.io%2Ftutorials%2Fdata_loading.html%23write-a-custom-dataloader&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150368167&sdata=N3BRcKXz9QhK5CkGNQG74msjjMSf1xXKynhi3K2NpV8%3D&reserved=0 to handle rotated boxes. Some notable changes needed in the mapper:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU6MQNUAZHDMAM2H66DRGBDDHA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7PGDQ%23issuecomment-595522318&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150378165&sdata=6hTd03x22fkXHHU8lTETKVu1Q%2B%2B1etGCMw9%2Bw6NlKY0%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU3KD6LSQ7AKGJ3KY53RGBDDHANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150378165&sdata=lle9RdHjc%2FjWrXJZ8bS%2BzpGdOHoaXWcsv0xyo9oS7TE%3D&reserved=0.

st7ma784 commented 4 years ago

My training now works, cheers @ppwwyyxx!

for anyone else struggling, heres the cell that made a difference:

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
from detectron2.modeling import build_model
from detectron2.evaluation import inference_on_dataset, RotatedCOCOEvaluator
from detectron2 import model_zoo
from detectron2.data import transforms as T
from detectron2.data import detection_utils as utils
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader,build_detection_train_loader
import copy
from detectron2.engine import DefaultTrainer, default_argument_parser, default_setup, hooks, launch

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo
cfg.DATASETS.TRAIN = (['Train'])
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.MASK_ON=False
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.CHECKPOINT_PERIOD=2000
cfg.SOLVER.BASE_LR = 0.0005  # pick a good LR
cfg.SOLVER.MAX_ITER=10000   # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES =ClassCount  
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"
cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (10,10,5,5,1)
cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[-90,-60,-30,0,30,60,90]]
cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256 # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)#

def my_transform_instance_annotations(annotation, transforms, image_size, *, keypoint_hflip_indices=None):
  annotation["bbox"] = transforms.apply_rotated_box(np.asarray([annotation['bbox']]))[0]
  annotation["bbox_mode"] = BoxMode.XYXY_ABS
  return annotation

def mapper(dataset_dict):
  dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
  image = utils.read_image(dataset_dict["file_name"], format="BGR")
  image, transforms = T.apply_transform_gens([T.Resize((800, 800))], image)
  dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))
  annos = [
      my_transform_instance_annotations(obj, transforms, image.shape[:2])  
      for obj in dataset_dict.pop("annotations")
      if obj.get("iscrowd", 0) == 0
  ]
  instances = utils.annotations_to_instances_rotated(annos, image.shape[:2])
  dataset_dict["instances"] = utils.filter_empty_instances(instances)
  return dataset_dict

class MyTrainer(DefaultTrainer):
  @classmethod
  def build_evaluator(cls, cfg, dataset_name):
      output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
      evaluators = [RotatedCOCOEvaluator(dataset_name, cfg, True, output_folder)]
      return DatasetEvaluators(evaluators)
  @classmethod
  def build_train_loader(cls, cfg):
        return build_detection_train_loader(cfg,mapper=mapper)

EDIT Changed evaluator to ROTATEDcocoevaluator

kakaluote commented 4 years ago

My training is working，but it seem not convergence

breznak commented 4 years ago

@st7ma784 could you reformat your post with code (tripe `) to be more readable? Also, would you like to commit the rotated dataloader for the PR #996 ?

st7ma784 commented 4 years ago

@breznak sure- a little new to gtihub with not just my own project so not sure how. (this is one of my first "real world" projects using github)

st7ma784 commented 4 years ago

@kakaluote I found the same, with a very low LR, (I was using 0.0005) after 6000 iters there was no convergence, upping it to 0.005 helped massively, but I suspect its to do with the RegBox weights having angle as least. I think that might depend on the object shapes in the dataset, if your object are long and thin (Im using Ships) then the angle is far more important than the X\Y of a BBOX proposal. try changing cfg.MODEL.RPN.BBOX_REG_WEIGHTS or cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS back to (1,1,1,1,1)

Amir22010 commented 4 years ago

@st7ma784 you solved my training issue thanks for help:))))))))

i use your custom mapper function and modify as per your cfg params

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True) trainer = MyTrainer(cfg) trainer.resume_or_load(resume=False) trainer.train()

now i my training has started

[03/06 17:02:15 d2.utils.events]: eta: 1:31:35 iter: 799 total_loss: 0.204 loss_cls: 0.055 loss_box_reg: 0.057 loss_rpn_cls: 0.030 loss_rpn_loc: 0.042 time: 0.5960 data_time: 0.0080 lr: 0.000400 max_mem: 3370M [03/06 17:02:28 d2.utils.events]: eta: 1:31:26 iter: 819 total_loss: 0.252 loss_cls: 0.071 loss_box_reg: 0.079 loss_rpn_cls: 0.032 loss_rpn_loc: 0.071 time: 0.5963 data_time: 0.0067 lr: 0.000410 max_mem: 3370M [03/06 17:02:40 d2.utils.events]: eta: 1:31:17 iter: 839 total_loss: 0.307 loss_cls: 0.081 loss_box_reg: 0.097 loss_rpn_cls: 0.026 loss_rpn_loc: 0.073 time: 0.5965 data_time: 0.0089 lr: 0.000420 max_mem: 3370M [03/06 17:02:52 d2.utils.events]: eta: 1:31:06 iter: 859 total_loss: 0.298 loss_cls: 0.094 loss_box_reg: 0.096 loss_rpn_cls: 0.027 loss_rpn_loc: 0.055 time: 0.5968 data_time: 0.0076 lr: 0.000430 max_mem: 3370M

Amir22010 commented 4 years ago

@kakaluote I found the same, with a very low LR, (I was using 0.0005) after 6000 iters there was no convergence, upping it to 0.005 helped massively, but I suspect its to do with the RegBox weights having angle as least. I think that might depend on the object shapes in the dataset, if your object are long and thin (Im using Ships) then the angle is far more important than the X\Y of a BBOX proposal. try changing cfg.MODEL.RPN.BBOX_REG_WEIGHTS or cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS back to (1,1,1,1,1)

can i change both the param to (1,1,1,1,1)???

st7ma784 commented 4 years ago

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍 

import random
dataset_dicts = returnTRAINRECORDS()
print(ClassesTotals)
class myVisualizer(Visualizer):

    def draw_dataset_dict(self, dic):
        annos = dic.get("annotations", None)
        if annos:
            if "segmentation" in annos[0]:
                masks = [x["segmentation"] for x in annos]
            else:
                masks = None
            if "keypoints" in annos[0]:
                keypts = [x["keypoints"] for x in annos]
                keypts = np.array(keypts).reshape(len(annos), -1, 3)
            else:
                keypts = None

            boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]

            labels = [x["category_id"] for x in annos]
            names = self.metadata.get("thing_classes", None)
            if names:
                labels = [names[i] for i in labels]
            labels = [
                "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")
                for i, a in zip(labels, annos)
            ]
            self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)

        sem_seg = dic.get("sem_seg", None)
        if sem_seg is None and "sem_seg_file_name" in dic:
            sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)
        if sem_seg is not None:
            self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)
        return self.output
for d in random.sample(dataset_dicts, 3):
    e=dict(d)
    name=e.get("file_name")
    print(name)
    print(e.items())
    img = cv2.imread(name)
    visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)
    vis = visualizer.draw_dataset_dict(e, )
    cv2_imshow(vis.get_image()[:, :, ::-1])
Amir22010 commented 4 years ago

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍

import random
dataset_dicts = returnTRAINRECORDS()
print(ClassesTotals)
class myVisualizer(Visualizer):

    def draw_dataset_dict(self, dic):
        annos = dic.get("annotations", None)
        if annos:
            if "segmentation" in annos[0]:
                masks = [x["segmentation"] for x in annos]
            else:
                masks = None
            if "keypoints" in annos[0]:
                keypts = [x["keypoints"] for x in annos]
                keypts = np.array(keypts).reshape(len(annos), -1, 3)
            else:
                keypts = None

            boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]

            labels = [x["category_id"] for x in annos]
            names = self.metadata.get("thing_classes", None)
            if names:
                labels = [names[i] for i in labels]
            labels = [
                "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")
                for i, a in zip(labels, annos)
            ]
            self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)

        sem_seg = dic.get("sem_seg", None)
        if sem_seg is None and "sem_seg_file_name" in dic:
            sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)
        if sem_seg is not None:
            self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)
        return self.output
for d in random.sample(dataset_dicts, 3):
    e=dict(d)
    name=e.get("file_name")
    print(name)
    print(e.items())
    img = cv2.imread(name)
    visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)
    vis = visualizer.draw_dataset_dict(e, )
    cv2_imshow(vis.get_image()[:, :, ::-1])

its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model,,,,,,,,i think it will be annotation["bbox_mode"] = BoxMode.XYWHA_ABS in your my_transform_instance_annotations()

st7ma784 commented 4 years ago

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle.

For reference, that code is used after I load in my dataset so I can verify its correct pre training.

From: Amir Khan notifications@github.com Sent: Friday, March 6, 2020 7:23:32 PM To: facebookresearch/detectron2 detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.uk; Mention mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍

import random

dataset_dicts = returnTRAINRECORDS()

print(ClassesTotals)

class myVisualizer(Visualizer):

def draw_dataset_dict(self, dic):

    annos = dic.get("annotations", None)

    if annos:

        if "segmentation" in annos[0]:

            masks = [x["segmentation"] for x in annos]

        else:

            masks = None

        if "keypoints" in annos[0]:

            keypts = [x["keypoints"] for x in annos]

            keypts = np.array(keypts).reshape(len(annos), -1, 3)

        else:

            keypts = None

        boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]

        labels = [x["category_id"] for x in annos]

        names = self.metadata.get("thing_classes", None)

        if names:

            labels = [names[i] for i in labels]

        labels = [

            "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")

            for i, a in zip(labels, annos)

        ]

        self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)

    sem_seg = dic.get("sem_seg", None)

    if sem_seg is None and "sem_seg_file_name" in dic:

        sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)

    if sem_seg is not None:

        self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)

    return self.output

for d in random.sample(dataset_dicts, 3):

e=dict(d)

name=e.get("file_name")

print(name)

print(e.items())

img = cv2.imread(name)

visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)

vis = visualizer.draw_dataset_dict(e, )

cv2_imshow(vis.get_image()[:, :, ::-1])

its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0.

Amir22010 commented 4 years ago

My training now works, cheers @ppwwyyxx!

for anyone else struggling, heres the cell that made a difference:

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
from detectron2.modeling import build_model
from detectron2.evaluation import inference_on_dataset, COCOEvaluator
from detectron2 import model_zoo
from detectron2.data import transforms as T
from detectron2.data import detection_utils as utils
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader,build_detection_train_loader
import copy
from detectron2.engine import DefaultTrainer, default_argument_parser, default_setup, hooks, launch

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo
cfg.DATASETS.TRAIN = (['Train'])
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.MASK_ON=False
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.CHECKPOINT_PERIOD=2000
cfg.SOLVER.BASE_LR = 0.0005  # pick a good LR
cfg.SOLVER.MAX_ITER=10000   # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES =ClassCount  
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"
cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (10,10,5,5,1)
cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[-90,-60,-30,0,30,60,90]]
cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256 # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)#

def my_transform_instance_annotations(annotation, transforms, image_size, *, keypoint_hflip_indices=None):
  annotation["bbox"] = transforms.apply_rotated_box(np.asarray([annotation['bbox']]))[0]
  annotation["bbox_mode"] = BoxMode.XYXY_ABS
  return annotation

def mapper(dataset_dict):
  dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
  image = utils.read_image(dataset_dict["file_name"], format="BGR")
  image, transforms = T.apply_transform_gens([T.Resize((800, 800))], image)
  dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))
  annos = [
      my_transform_instance_annotations(obj, transforms, image.shape[:2])  
      for obj in dataset_dict.pop("annotations")
      if obj.get("iscrowd", 0) == 0
  ]
  instances = utils.annotations_to_instances_rotated(annos, image.shape[:2])
  dataset_dict["instances"] = utils.filter_empty_instances(instances)
  return dataset_dict

class MyTrainer(DefaultTrainer):
  @classmethod
  def build_evaluator(cls, cfg, dataset_name):
      output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
      evaluators = [COCOEvaluator(dataset_name, cfg, True, output_folder)]
      return DatasetEvaluators(evaluators)
  @classmethod
  def build_train_loader(cls, cfg):
        return build_detection_train_loader(cfg,mapper=mapper)

,i think it will be annotation["bbox_mode"] = BoxMode.XYWHA_ABS in your my_transform_instance_annotations()

Amir22010 commented 4 years ago

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle. For reference, that code is used after I load in my dataset so I can verify its correct pre training. … ____ From: Amir Khan notifications@github.com Sent: Friday, March 6, 2020 7:23:32 PM To: facebookresearch/detectron2 detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.uk; Mention mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21) This email originated outside the University. Check before clicking links or attachments. I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍 import random dataset_dicts = returnTRAINRECORDS() print(ClassesTotals) class myVisualizer(Visualizer): def draw_dataset_dict(self, dic): annos = dic.get("annotations", None) if annos: if "segmentation" in annos[0]: masks = [x["segmentation"] for x in annos] else: masks = None if "keypoints" in annos[0]: keypts = [x["keypoints"] for x in annos] keypts = np.array(keypts).reshape(len(annos), -1, 3) else: keypts = None boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos] labels = [x["category_id"] for x in annos] names = self.metadata.get("thing_classes", None) if names: labels = [names[i] for i in labels] labels = [ "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "") for i, a in zip(labels, annos) ] self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts) sem_seg = dic.get("sem_seg", None) if sem_seg is None and "sem_seg_file_name" in dic: sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE) if sem_seg is not None: self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5) return self.output for d in random.sample(dataset_dicts, 3): e=dict(d) name=e.get("file_name") print(name) print(e.items()) img = cv2.imread(name) visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1) vis = visualizer.draw_dataset_dict(e, ) cv2_imshow(vis.get_image()[:, :, ::-1]) its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0.

so, do u know which pre saved model has boxmode XYWHA_ABS as output???,,,or i have to train it from scratch

st7ma784 commented 4 years ago

Train from scratch I’m afraid. When a checkpoint file is loaded in there’s often a

‘’’ 'proposal_generator.anchor_generator.cell_anchors.0' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped. 'proposal_generator.anchor_generator.cell_anchors.1' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped. 'proposal_generator.anchor_generator.cell_anchors.2' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped. 'proposal_generator.anchor_generator.cell_anchors.3' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped. 'proposal_generator.anchor_generator.cell_anchors.4' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped. ‘’’ Meaning the model is using a different set of layer shapes then the pretrained bit.

Coincidently, this is why colab is such a great environment, the access to decent GPUs for short stints of training with transferred learning.

From: Amir Khan notifications@github.com Sent: Friday, March 6, 2020 7:46:03 PM To: facebookresearch/detectron2 detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.uk; Mention mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle. For reference, that code is used after I load in my dataset so I can verify its correct pre training. … ____ From: Amir Khan notifications@github.commailto:notifications@github.com Sent: Friday, March 6, 2020 7:23:32 PM To: facebookresearch/detectron2 detectron2@noreply.github.commailto:detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.ukmailto:s.mander@lancaster.ac.uk; Mention mention@noreply.github.commailto:mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806448410&sdata=pvk6aJxWyApuq%2BFz2IE5XhKrY5mxq21cThCn9wBFYug%3D&reserved=0) This email originated outside the University. Check before clicking links or attachments. I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍 import random dataset_dicts = returnTRAINRECORDS() print(ClassesTotals) class myVisualizer(Visualizer): def draw_dataset_dict(self, dic): annos = dic.get("annotations", None) if annos: if "segmentation" in annos[0]: masks = [x["segmentation"] for x in annos] else: masks = None if "keypoints" in annos[0]: keypts = [x["keypoints"] for x in annos] keypts = np.array(keypts).reshape(len(annos), -1, 3) else: keypts = None boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos] labels = [x["category_id"] for x in annos] names = self.metadata.get("thing_classes", None) if names: labels = [names[i] for i in labels] labels = [ "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "") for i, a in zip(labels, annos) ] self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts) sem_seg = dic.get("sem_seg", None) if sem_seg is None and "sem_seg_file_name" in dic: sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE) if sem_seg is not None: self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5) return self.output for d in random.sample(dataset_dicts, 3): e=dict(d) name=e.get("file_name") print(name) print(e.items()) img = cv2.imread(name) visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1) vis = visualizer.draw_dataset_dict(e, ) cv2_imshow(vis.get_image()[:, :, ::-1]) its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806448410&sdata=UThDr9DIcEaMXhRW%2B7xHODMXGLQXg2VsYExY7eHN%2BRY%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806458408&sdata=7%2FJd%2FxN1uFZmvy1OodmR3U2JStzr398lY7WiToF8brU%3D&reserved=0.

so, do u know which pre saved model has boxmode XYWHA_ABS as output???,,,or i have to train it from scratch

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TUYUTJYIBI3G53PHVUTRGFHHXA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCS7DI%23issuecomment-595931021&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806458408&sdata=2dwl2wOJeAf29XFS8pH009o3JL%2FFZ1yoQx5JnySup9c%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TUZUEZ55AJYFHVS6COLRGFHHXANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806468404&sdata=xWu%2FdUVqz%2F5qJCl45HwWsWOaLVG1KwL4Osh5DHir8eo%3D&reserved=0.

kakaluote commented 4 years ago

@st7ma784 Could you share the RotatedCOCOEvaluator？

st7ma784 commented 4 years ago

I think the code you’re after is: ‘’’ from detectron2.evaluation import RotatedCOCOEvaluator ‘’’

Hope that helps

From: kakaluote notifications@github.com Sent: Monday, March 9, 2020 7:51:27 AM To: facebookresearch/detectron2 detectron2@noreply.github.com Cc: Mander, Stephen (Student) s.mander@lancaster.ac.uk; Mention mention@noreply.github.com Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

@st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942416625&sdata=QPFYZ7A7cDEHIemGr%2F0ojpVp87mJ7dhkxnoOCEm148c%3D&reserved=0 Could you share the RotatedCOCOEvaluator？

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TUZQEXLUT2ZOI7UOOHDRGSNX7A5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOGAJRI%23issuecomment-596378821&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942426619&sdata=C2uPc%2BWrg54oenV7hwqmQAfuetuUHtWUASIotoJ5V2M%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU6TPYUZBZQ5BTQV6DLRGSNX7ANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942426619&sdata=dsWVgkLnO41kBiVNrywUsilPRF3g2%2BUA%2Bk5suOGdnJg%3D&reserved=0.