Open jvdgoltz opened 2 years ago
github-actions[bot]
commented
2 years ago You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template. The following information is missing: "Instructions To Reproduce the Issue and Full Logs";
jvdgoltz
commented
2 years ago I was looking more closely at this part of the stack trace:
File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/proposal_generator/rrpn.py", line 176, in label_and_sample_anchors
match_quality_matrix = retry_if_cuda_oom(pairwise_iou_rotated)(gt_boxes_i, anchors)As a workaround, I could try implement my own RRPN using a custom retry_if_cuda_oom:
@contextmanager
def _ignore_torch_cuda_oom():
"""
A context which ignores CUDA OOM exception from pytorch.
"""
try:
yield
except RuntimeError as e:
# NOTE: the string may change?
if "CUDA out of memory. " in str(e):
pass
elif "CUDA error: an illegal memory access was encountered" in str(e):
pass
else:
raiseI will also investigate if gt_boxes_i, anchors have weird values!
Would love to hear your opinions!
ppwwyyxx
commented
2 years ago From the given information it does suggest a potential bug in box_iou_rotated, especially given the large image sizes I suspect something is out of bound. But we do not know how it happened.
The best way to help report the issue is to catch the error and save the input data that causes box_iou_rotated to fail.
If you can then, in a short script, call box_iou_rotated with the bad data and reproduce the issue, provide the script and the data and I'm sure it can be fixed.
Mark-C-Lowell
commented
2 years ago We encountered this problem as well. We caught the boxes and pickled them, and it appears to be caused by the product of the number of boxes being too large. We can replicate the illegal memory access using:
import torch
from detectron2.layers.rotated_boxes import pairwise_iou_rotated
M = 2 * 1024
N = 1024**2
boxes_1 = torch.ones((M, 5), device='cuda')
boxes_2 = torch.ones((N, 5), device='cuda')
pairwise_iou_rotated(boxes_1, boxes_2)
torch.cuda.synchronize()If we reduce M or N by one, the illegal memory access no longer occurs.
jvdgoltz
commented
2 years ago @Mark-C-Lowell Good job debugging the issue! Saved me some hassle ;) It totally makes sense: The combination of super high resolution and rotated bounding boxers results in many more anchors which is where our issue appears.
We work around it by implementing a strided anchor generator and RPN head (something that was on our roadmap anyway) that reduces the number of anchors significantly
ppwwyyxx
commented
2 years ago I cannot reproduce the error using code above - it runs correctly. This problem may be specific to certain environment.
Mark-C-Lowell
commented
2 years ago Sorry, I had a typo in my comment. It errors out when $M * N$ exceeds $2 \cdot 1024^3$, not when it equals it.
Hello all, really enjoy
detectron2and its possibilities for customization but I encountered a problem that makes me scratch my head:Full runnable code or full changes you made: I trained https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml with the configuration for rotated boxes that is based on https://github.com/facebookresearch/detectron2/issues/21#issuecomment-595522318:
What exact command you run: Custom
train.pyscript for initializing the config, but very similar totrain_net.py, after issue appeared for the first time, I also setCUDA_LAUNCH_BLOCKING=1Full logs or other relevant observations:
After a few hundred/thousand iterations my training script throws an exception.
I encountered this for the first time when switching my config from a aligned box detection to rotated box detection. Additionally, you should know that we are aspiring to train with a very high image resolution (eventually 8000x5000 pixels). In that config, I was using (Don't have exact
envbut this should give all the info needed):At first I thought it was a memory issue, even though I still had some free memory according to
nvidia-smi. So I reduced the amount of proposals by reducing the number of aspect rations and angles (seeconfigabove) and the problem disappeared. I was not too happy with that because I couldn't make full use of the GPU.Instead, we scaled up to a
4 x V100setup (so training with DDP and 4 workers) to increase the input resolution to the above values. I tried to add back an additional value for the aspect ratio, and that brought back theIllegal memory accessagain. So I reverted and it worked again.Recently, we switched to
2 x A100GPUs. For that I had to use CUDA 11, and now the error is coming back even though I didn't increase the number of proposals or input resolution.Environment:
This is the environment for the most recent configuration that I used when I encountered this issue.
So this error happened on two different GPU platforms with two different CUDA versions and it happened always in the
_C.box_iou_rotated(boxes1, boxes2). I never had this problem when training on the aligned boxes. It could be a CUDA bug, but because it is so specific to rotated boxes I came here first.Help is greatly appreciated!