search

facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.
https://detectron2.readthedocs.io/en/latest/
Apache License 2.0
30.33k stars 7.46k forks source link

Extraction of Region Proposals is not deterministic #2480

Closed varunnrao closed 3 years ago

varunnrao commented 3 years ago

Instructions To Reproduce the Issue:

  1. Full runnable code or full changes you made: 
    
plt.rcParams['figure.figsize'] = [12, 8]
plt.rcParams['figure.dpi'] = 100
CONFIG_PATH = "config.yaml"
MODEL_PATH = "model_final.pth"

cfg = get_cfg() cfg.merge_from_file(CONFIG_PATH) cfg.MODEL.WEIGHTS = MODEL_PATH model = build_model(cfg) model.eval() transform_gen = T.ResizeShortestEdge( [cfg.INPUT.MIN_SIZE_TEST, cfg.INPUT.MIN_SIZE_TEST], cfg.INPUT.MAX_SIZE_TEST ) input_format = cfg.INPUT.FORMAT device = torch.device(cfg.MODEL.DEVICE) num_channels = len(cfg.MODEL.PIXEL_MEAN) pixel_mean = torch.Tensor(cfg.MODEL.PIXEL_MEAN).view(num_channels, 1, 1) pixel_std = torch.Tensor(cfg.MODEL.PIXEL_STD).view(num_channels, 1, 1) normalizer = lambda x: (x - pixel_mean) / pixel_std

img = cv2.imread('image0008.jpg') plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))

bbox_list = [] logit_list = [] if(input_format == "RGB"):

whether the model expects BGR inputs or RGB

img = img[:, :, ::-1]

input_height, input_width = img.shape[:2] img_transform = transform_gen.get_transform(img).apply_image(img) img_tensor = torch.as_tensor(img_transform.astype("float32").transpose(2, 0, 1)) img_tensor = normalizer(img_tensor) img_gpu = img_tensor.to(device)

can be a list of images;

img_list = ImageList.from_tensors([img_gpu], model.backbone.size_divisibility)

features = model.backbone(img_list.tensor)

refer https://detectron2.readthedocs.io/tutorials/models.html#model-output-format

for output format of proposal generator

returns a list of proposals;

proposals, _ = model.proposal_generator(img_list, features) proposal_boxes = proposals[0].proposal_boxes scale_x, scale_y = (input_width / proposals[0].image_size[1], input_height / proposals[0].image_size[0]) proposal_boxes.scale(scale_x, scale_y)

note output format is in [XYXY] even though input may be in [XYXY] or [XYWH]

objectness_logits = proposals[0].objectness_logits

for bbox, logit in zip(proposal_boxes, objectness_logits): bbox, logit = bbox.cpu().numpy(), logit.cpu().numpy().item() x, y, w, h = int(bbox[0]), int(bbox[1]), int(bbox[2]) - int(bbox[0]) + 1, int(bbox[3]) - int(bbox[1]) + 1 plt.gca().add_patch(Rectangle((x,y),w,h,linewidth=1,edgecolor='r',facecolor='none')) bbox_list.append([x,y,w,h]) logit_list.append(logit) plt.show() 

2. What exact command you run:
I ran this script
3. __Full logs__ you observed:
Output list `bbox_list` and `logit_list` differ for the same image when I execute the script multiple times. 
Further it is observed that the bbox coordinates obtained when `model(inputs)` (the entire model inference rather than just the proposal generator) is executed or the `DefaultPredictor` is used, are most likely not similar to that obtained at the RPN stage. 

## Expected behavior:
I would expect the bbox coordinates are constant for different execution runs when the image, config and model remains the same. However, each execution run differs in the output.

- Are there specific pre/post processing steps missing?
- Is the output of the RPN stage expected to be non-deterministic? 
- My goal is to use the output of the RPN (image crops) to build another classifier. So I'd require the outputs of RPN to be deterministic. Is this possible?

## Environment:

Provide your environment information using the following command:

wget -nc -q https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py

wget: /home/ubuntu/anaconda3/lib/libuuid.so.1: no version information available (required by wget)

sys.platform linux Python 3.6.12 Anaconda, Inc. (default, Sep 8 2020, 23:10:56) [GCC 7.3.0] numpy 1.19.2 detectron2 0.1.1 detectron2 compiler GCC 5.4 detectron2 CUDA compiler 10.0 detectron2 arch flags sm_37 DETECTRON2_ENV_MODULE PyTorch 1.4.0 PyTorch debug build False CUDA available True GPU 0,1,2,3,4,5,6,7 Tesla K80 CUDA_HOME /usr/local/cuda-10.0 NVCC Cuda compilation tools, release 10.0, V10.0.130 Pillow 8.0.1 torchvision 0.5.0 torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 cv2 3.4.2

PyTorch built with:

ppwwyyxx commented 3 years ago

PyTorch is not deterministic: https://pytorch.org/docs/stable/notes/randomness.html. Therefore there is probably not much we can do, unless there is evidence that detectron2 introduces more non-determinism than what's in pytorch.

varunnrao commented 3 years ago

I don't think it's the pytorch non determinism causing the issue here. I am using a pretrained detectron2 masked cascade faster rcnn model for inference.

It doesn't seem right that the outputs differ at the RPN stage but it's reproducible after the RoI heads.

Hence I feel there is possibly some non determinism introduced by detectron2. Could you please have another look at the code, in case there is some obvious mistakes in the pre/post processing or model loading?

Could you please reopen this issue? It doesn't seem resolved. Thanks.