How to undistort image before detecting objects?

[Nangsaga]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello,

I am using a camera with FOV angle of 120 deg to detect objects using a model trained by YOLOV5. The camera captures the object images with large distortion at short distance. I calibrated the camera using opencv and got camera matrix and distortion coefficients. However, I do not know how to incorporate the camera matrix and distortion coefficients into the detect.py. Any one please kindly advise me how to modify detect.py of yolov5 to undistort images before feeding them into object detector. Many thanks.

Additional

No response

[github-actions[bot]]

👋 Hello @Nangsaga, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@Nangsaga hi there!

To undistort images before feeding them into the YOLOv5 object detector, you can use the OpenCV undistort() function. Here's an example of how you can modify the detect.py script to incorporate the camera matrix and distortion coefficients:

1. Import the necessary libraries:

   import cv2
   import numpy as np

2. Load the camera matrix and distortion coefficients:

   # Replace `<camera_matrix_file>` and `<distortion_coefficients_file>` with the respective file paths
   camera_matrix = np.load('<camera_matrix_file>.npy')
   dist_coefficients = np.load('<distortion_coefficients_file>.npy')

3. Modify the main loop of the script to undistort images:

   for path, img, im0s, _ in dataset:
   # Undistort the image
   h, w = im0s.shape[:2]
   new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(camera_matrix, dist_coefficients, (w, h), 1, (w, h))
   undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix)
   
   # Proceed with object detection on the undistorted image
   # ...

Please make sure to replace <camera_matrix_file> and <distortion_coefficients_file> with the actual file paths where you have saved the camera matrix and distortion coefficients.

Hope this helps! Let me know if you have any further questions.

[Nangsaga]

Hi Glenn Jocher, Thank you for quick response. I tried following your instructions and received error when running detect.py as below h, w = im0s.shape[:2] AttributeError: 'list' object has no attribute 'shape'

I tried to explicitly convert h and w to NumPy arrays using np.array( ) but still got the error. Could please further guide me to solve the problem. I am newbie in python programming.

[glenn-jocher]

@Nangsaga hi there,

Thank you for trying out the instructions. I apologize for the mistake in my previous response. It seems that im0s is a list object instead of a NumPy array, which is causing the error.

To fix this issue, you can modify the code by converting im0s to a NumPy array before accessing its shape. Here's the modified code snippet:

for path, img, im0s, _ in dataset:
    # Convert im0s to a NumPy array
    im0s = np.array(im0s)

    # Undistort the image
    h, w = im0s.shape[:2]
    new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(camera_matrix, dist_coefficients, (w, h), 1, (w, h))
    undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix)

    # Proceed with object detection on the undistorted image
    # ...

Please give it a try and let me know if you encounter any further issues. I'm here to help.

Best regards, Glenn Jocher

[Nangsaga]

Thank you very much, Glenn. After revising the detect.py, I gave a try with the webcam of my laptop PC and received error as below. It seemed that im0s was empty. I also tried other source = 1 or 2 (no camera connected) it failed to open camera.

python detect.py --source 0 --weight best.pt yolov5\detect.py", line 127, in run undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix) cv2.error: OpenCV(4.5.3) C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-1i5nllza\opencv\modules\core\src\matrix.cpp:250: error: (-215:Assertion failed) s >= 0 in function 'cv::setSize'

python detect.py --source 1 --weight best.pt yolov5\utils\dataloaders.py", line 341, in init assert cap.isOpened(), f'{st}Failed to open {s}' AssertionError: 1/1: 1... Failed to open 1

[glenn-jocher]

@Nangsaga

I apologize for the inconvenience you are experiencing. Let's address the issues you encountered one by one:

1. Error with im0s being empty: The error message suggests that im0s is empty, which could happen if the source you specified (--source 0) is unable to capture frames from the webcam. Please ensure that your webcam is properly connected and accessible by other applications on your laptop. You can also try specifying a different source, such as --source 1 or --source 2, to see if the issue persists.

2. Error with opening the camera source: The error message AssertionError: 1/1: 1... Failed to open 1 indicates that the camera source 1 could not be opened. In this case, 1 represents the default webcam. Please make sure that the correct camera index is being used by verifying the index of the webcam connected to your laptop. You can try different source indices (0, 1, 2, etc.) to identify the correct index for your webcam.

If these suggestions do not resolve the issues, please provide additional details, such as the operating system you are using and any other relevant information that may help us further troubleshoot the problem.

We appreciate your patience, and we are here to assist you in resolving these issues.

Kind regards, Glenn Jocher

[Nangsaga]

Thanks Glenn for your continuous supports. I do not know why detect.py could not open the built-in webcam of my laptop. I will try with USB camera next week and report any result to you.

[glenn-jocher]

@Nangsaga hi there,

I'm glad to hear that you found the support helpful. It's possible that there might be some compatibility issues with the built-in webcam on your laptop. Trying with a USB camera would be a good next step to see if the issue persists. Please feel free to report back with the results. If you have any further questions or need assistance, please don't hesitate to ask.

Best regards, Glenn Jocher

[Nangsaga]

Hi Glenn, I tried the modified detect.py with USB webcam but still received the error of empty im0s Any suggestions? Thank you for your supports.

[glenn-jocher]

@Nangsaga hi there,

I apologize for the continued issues you're experiencing. If you're still encountering the empty im0s error with the modified detect.py script when using a USB webcam, there are a few possible reasons for this:

1. Camera Compatibility: Make sure that the USB webcam you are using is compatible with OpenCV and is properly connected to your system. You can try using the webcam with other applications or testing it on a different system to verify its functionality.

2. Camera Source: Ensure that you are providing the correct camera source index when running the detect.py script. You can try different source indices (e.g., 0, 1, 2, etc.) to see if the issue is related to the camera source.

3. Permissions: Check that your system has proper permissions to access and use the USB webcam. This can sometimes be an issue, especially on certain operating systems or with specific camera models. You may need to grant permissions or adjust settings to allow camera access.

If none of these suggestions resolve the issue, it might be helpful to provide additional information, such as the operating system you are using, the exact error message you're seeing, and any other relevant details about the USB webcam (make, model, etc.).

I hope this helps! If you have any further questions or need additional assistance, please let me know.

[Nangsaga]

Thank you very much for your continuous supports. I have successfully detected objects with the same USB webcams using the original detect.py (unmodified). Belows are mofidied codes of the detect.py, which resulted in empty im0s.

`# Run inference model.warmup(imgsz=(1 if pt else bs, 3, *imgsz)) # warmup seen, windows, dt = 0, [], (Profile(), Profile(), Profile())

# Replace `<camera_matrix_file>` and `<distortion_coefficients_file>` with the respective file paths
camera_matrix = np.loadtxt('../yolov5/camera/mtx.csv', delimiter=',')
dist_coefficients = np.loadtxt('../yolov5/camera/dist.csv', delimiter=',')

for path, im, im0s, vid_cap, s in dataset:
    # Undistort the image
    im0s = np.asarray(im0s)
    h, w = im0s.shape[:2]
    new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(camera_matrix, dist_coefficients, (w, h), 1, (w, h))
    undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix)

    with dt[0]:
        im = torch.from_numpy(im).to(device)
        im = im.half() if model.fp16 else im.float()  # uint8 to fp16/32
        im /= 255  # 0 - 255 to 0.0 - 1.0
        if len(im.shape) == 3:
            im = im[None]  # expand for batch dim

    # Inference
    with dt[1]:
        visualize = increment_path(save_dir / Path(path).stem, mkdir=True) if visualize else False
        pred = model(im, augment=augment, visualize=visualize)

    # NMS
    with dt[2]:
        pred = non_max_suppression(pred, conf_thres, iou_thres, classes, agnostic_nms, max_det=max_det)
    # Second-stage classifier (optional)
    # pred = utils.general.apply_classifier(pred, classifier_model, im, im0s)

    # Process predictions
    for i, det in enumerate(pred):  # per image
        seen += 1
        if webcam:  # batch_size >= 1
            p, im0, frame = path[i], undistorted_img[i].copy(), dataset.count
            s += f'{i}: '
        else:
            p, im0, frame = path, undistorted_img.copy(), getattr(dataset, 'frame', 0)`

If possible, please check whether there is something wrong with the modified part.

[glenn-jocher]

@Nangsaga hello,

I appreciate your continuous support and feedback. Based on the modified code you provided, it seems that the issue may lie in the following lines:

im0s = np.asarray(im0s)
h, w = im0s.shape[:2]
new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(camera_matrix, dist_coefficients, (w, h), 1, (w, h))
undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix)

It's worth double-checking the following points:

1. Ensure that the camera matrix file (mtx.csv) and the distortion coefficients file (dist.csv) are located in the correct paths (../yolov5/camera/) relative to the detect.py script.
2. Verify that the camera matrix and distortion coefficients files contain the correct values for your specific camera setup.
3. Make sure that the camera matrix and distortion coefficients files are in the correct format (e.g., comma-separated values in the correct order).

If you have confirmed these points and are still encountering issues, it may be beneficial to provide more details about the specific error message or behavior you are experiencing. This will help in further identifying the problem.

I hope this helps you troubleshoot the issue. If you have any additional questions or require further assistance, don't hesitate to ask.

[Nangsaga]

Many thanks, Glenn

The points 1-3 were OK. Here are the error messages when executed python detect2.py --source 1 --weights best.pt --save-txt --save-conf

PS C:\yolov5> python detect2.py --source 1 --weights best.pt --save-txt --save-conf detect2: weights=['best.pt'], source=1, data=data\coco128.yaml, imgsz=[640, 640], conf_thres=0.6, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=True, save_conf=True, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs\detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False YOLOv5 v6.2-21-g1cd3e752 Python-3.10.2 torch-1.13.0.dev20220818+cu116 CUDA:0 (NVIDIA GeForce RTX 3070 Laptop GPU, 8192MiB)

Fusing layers... YOLOv5l summary: 267 layers, 46194438 parameters, 0 gradients, 107.9 GFLOPs 1/1: 1... Success (inf frames 640x480 at 30.00 FPS)

Traceback (most recent call last): File "C:\yolov5\detect2.py", line 268, in main(opt) File "C:\yolov5\detect2.py", line 263, in main run(*vars(opt)) File "C:\Users\Nang\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(args, **kwargs) File "C:\yolov5\detect2.py", line 124, in run undistorted_img = cv2.undistort(im0s, camera_matrix, dist_coefficients, None, new_camera_matrix) cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\core\src\matrix.cpp:250: error: (-215:Assertion failed) s >= 0 in function 'cv::setSize'

[glenn-jocher]

@Nangsaga hi there,

Based on the error message you provided, it appears that there is an issue with resizing the image using cv2.undistort(). The specific error message indicates an "Assertion failed" related to the size of the image.

To further investigate and resolve this issue, here are a few steps you can try:

1. Ensure that the camera matrix and distortion coefficients you provided are correct and correspond to your specific camera setup.

2. Confirm that the im0s variable is being correctly passed to the cv2.undistort() function. You can print the shape of im0s just before the cv2.undistort() line to verify its dimensions.

3. Check the version of OpenCV you are using (4.5.5 in your case). It's possible that there may be compatibility issues between the version of OpenCV and the YOLOv5 codebase.

4. Consider updating your OpenCV version to a more recent one if feasible, or try using a different version to see if that resolves the issue.

Please let me know if the above suggestions help or if you require further assistance. I'm here to help you troubleshoot and resolve this issue.

[Nangsaga]

Hi there,

I went through 4 steps suggested by you but could not find the reason why im0s was no size.

[glenn-jocher]

@Nangsaga hi there,

I'm sorry to hear that you couldn't find the reason for the empty size of im0s, despite going through the suggested steps. To further investigate the issue, it would be helpful to gather more information. Could you please provide additional details such as the version of OpenCV you are using, the specific camera setup you have, and any other relevant information about your environment and configuration? This will assist in identifying the root cause of the problem.

Thank you, Your Name

[Nangsaga]

Hi Glenn,

1) I am using Logicool webcamera C930e and calibrated it using Opencv. dist.csv [2.823844316410053720e-02,-1.240660129546049539e-01,-4.767963915712831145e-03,5.052226739666718439e-03,1.487490095954806524e-01] mtx.csv [6.294229143590149533e+02,0.000000000000000000e+00,3.313938673974921016e+02 0.000000000000000000e+00,6.293268105103638845e+02,2.561931810901435256e+02 0.000000000000000000e+00,0.000000000000000000e+00,1.000000000000000000e+00]

2) no shape was printed

3-4) I updated Opencv to the latest version 4.8.1

Thank you.

[glenn-jocher]

@Nangsaga hi,

Thank you for providing the additional information about your camera setup and calibration. I understand that you're still experiencing issues with the empty size of im0s despite the updates and calibration.

To further investigate the problem, I recommend checking the following:

1) Ensure that the camera is functioning properly by testing it with other applications or tools outside of YOLOv5.

2) Verify that the camera calibration is accurate and that the provided dist.csv and mtx.csv files are correct for your specific camera. Double-check the values and the order in which they are provided in the files.

3) Review the code where im0s is passed to cv2.undistort() and check for any potential issues or mistakes in the implementation, such as incorrect variable assignments or data types.

If the issue persists, it would be helpful to provide more details about the specific version of YOLOv5 you are using, as well as any relevant sections of code where the camera input is being processed. Additionally, any error messages or stack traces related to this issue would be beneficial for further diagnosis.

Thank you for your cooperation, and please let me know if there's anything else I can assist you with.

Best regards, Your Name

[Nangsaga]

Thank you for your quick response.

1-2) The used camera and calibration data have worked well with Opencv application such as ArUco marker detection. The 3D estimation gave quite accurate results so I think dist.csv and mtx.csv are correct. 3) I will thoroughly review the code and report to you later.

[glenn-jocher]

@Nangsaga thank you for sharing the additional information.

Based on your response, it seems that the camera and calibration data are functioning correctly with other applications using OpenCV, which suggests that the issue may be specific to the YOLOv5 implementation.

I appreciate your commitment to reviewing the code thoroughly. Please take your time to investigate and analyze the code, and let us know if you discover any potential issues or mistakes. Sharing specific details or snippets of the code that you suspect may be causing the problem would be helpful for further analysis.

We look forward to your report and will be glad to assist you further with resolving this issue.

Thank you, Your Name

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐