RGB Image Kernel-Based Board Pose Detection

[MohammadKhan-3]

Goal:

Create an Image Kernel to pass over a frame from Intel Realse d435i camera to detect a blue, red, and green square.

• For this issue, I'll only focus on detecting the blue square since the detection process is the same for red and green.
• If unfamiliar with concept of image kernels & convolutions check out this site: https://setosa.io/ev/image-kernels/

Computer Environment:

• Ubuntu 18.04
• Python 2.7
• OpenCV: 4.2.0

Current State:

• Used cv2.filter2D() function with frame from RealSense and custom Kernel

  Kernel:

• Kernel is made by creating an empty numpy array (51x51) array
• Frame size is 480x640 px
• Since Blue is the target color the RGB values of [0,0,255] are appended into the 51x51 array
• I did try to use an image of the blue square as the kernel but the heat map came up blank shown below

Issue:

• The heatmap produces a square in the incorrect location.
• It should be in the same location as the square in the image on the left if it's properly detected.
• past attempts: I tried flipping the kernel using cv2.flip() but that did not fix the orientation because the correct orientation was reflected over the y-axis or x-axis outside of the positive quadrant.

Code:

## IMPORTS
import sys
import os

# Add game_scripts directory to the python-modules path options to allow importing other python scripts
# insert at 1, 0 is the script path (or '' in REPL)
# sys.path.insert(1, '//home/martinez737/tic-tac-toe_ws/src/tic_tac_toe/game_scripts')
ttt_pkg = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
path_2_game_scripts = ttt_pkg + '/game_scripts'
sys.path.insert(1, path_2_game_scripts)

import rospy
import tf2_ros
import cv2

# ROS Data Types
from sensor_msgs.msg import Image
from geometry_msgs.msg import TransformStamped

# Custom Tools
    # from Realsense_tools import *
from transformations import *
from shape_detector import *
from cv_bridge import CvBridge, CvBridgeError

# System Tools
import pyrealsense2 as rs
import time
from math import pi, radians, sqrt, atan
import numpy as np
import matplotlib.pyplot as plt 

def kernel_color_detect(image):
    '''
    purpose: slides an array (nxn pixels) across an image to find color
    - obtain heat map of the color
    - find pixel location
    - get transform
    '''     
    # Uncomment below when using camera feed
    frame = image.copy()
    # cv2.imshow('Frame',frame)
    # cv2.waitKey(0)

    # rows = len(frame)
    # cols = len(frame[0])
    # print(rows) # 480
    # print(cols) # 640

    # Uncomment below when using image
    # test_image= os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + '/sample_content/sample_images/CorrectedColoredSquares_Color.png'
    # frame = cv2.imread(test_image)

    # Uncomment for Debuggging Frame
    # print(frame)
    # size = frame.shape
    # print(size)

    # make matrix size 50x50 with rgb values inside
    # red = [255,0,0]
    # green = [0,255,0]
    # blue = [0,0,255]

    # Ref: https://www.geeksforgeeks.org/image-filtering-using-convolution-in-opencv/amp/
    # Plan as of 10/20/21: use read image into kernel matrix then perform convolutions 
    # GeeksforGeeks use Python 3 (we have Python 2.7)

    # Obtain directory for blue, green, red square crops
    blue_square_crop = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + '/sample_content/sample_images/blue_square_crop.png'
    green_square_crop = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + '/sample_content/sample_images/green_square_crop.png'
    red_square_crop = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + '/sample_content/sample_images/red_square_crop.png'

    # Numpy: uses (y,x) / OpenCv (x,y)
    blue_square = cv2.imread(blue_square_crop)

    # Uncomment: Checking to see if original frame being read is flipped
    # cv2.imshow('Blue Square',blue_square)
    # cv2.waitKey(0)

    kernel_blue = np.zeros((51,51))  # creating an empty array
    # Uncomment below to use RGB values as kernel
    kernel_blue = np.append(kernel_blue,[0,0,255])

    # Uncomment below to use image as kernel
    # kernel_blue = np.append(kernel_blue,blue_square)
    # print('kernel_blue:',kernel_blue)
    # kernel_blue = np.asanyarray(kernel_blue,np.float32)

    # kernel_blue = cv2.flip(kernel_blue,-5)
    # can't assume kernel to be symmetric matrix, need to flip it before passing it to filter2D
    # cv2.flip()
    # negative number: flips about both axes
    #  0: flip around x-axis
    # positive number: flip around y-axis
    # docs on cv2.flip: https://docs.opencv.org/4.5.3/d2/de8/group__core__array.html#gaca7be533e3dac7feb70fc60635adf441

    # Uncomment below for debugging
    # print('Kernel')
    # print(kernel_blue) # Also not finding the Kernel
    # Kernel size must by n x n where n- odd numbers
    # blue, green, and red square crops are 55 x 55 pixels
    blue_heatmap = cv2.filter2D(frame,-3,kernel_blue)
    blue_heatmap = cv2.applyColorMap(blue_heatmap,cv2.COLORMAP_HOT)
    # cv2.applyColorMap() changes color of heatmap

    print('Blue HeatMap: ',blue_heatmap)
    # opencv docs on filter2D:
    # https://docs.opencv.org/4.2.0/d4/d86/group__imgproc__filter.html#ga27c049795ce870216ddfb366086b5a04
    '''
    filter2D parameters:
        InputArray src, 
        OutputArray dst, 
        int ddepth, 
        InputArray kernel,
        Point anchor = Point(-1,-1),
        double delta = 0,
        int borderType = BORDER_DEFAULT   
    '''
    plt.figure(1) # Realsense Frame
    plt.imshow(cv2.cvtColor(frame,cv2.COLOR_BGR2RGB))
    plt.figure(2) # HeatMap Output
    plt.imshow(cv2.cvtColor(blue_heatmap,cv2.COLOR_BGR2RGB))
    plt.show()`

[MohammadKhan-3]

Flip it about about both horizontal axis to try to get the shape in the correct orientation.

• Flipping it about the x & y axis yields the same result

The correct orientation is below the x-axis.

Reason for Flipping Kernel:

• Can't assume kernel to be symmetric matrix, need to flip it before passing it to filter2D
• filter2D performs a correlation calculation
• If kernel matrix is symmetric then correlation = convolution --> we cannot assume a symmetric kernel matrix
• For asymmetric kernels: correlation ≠ convolution
• According to StackOverflow flipping the kernel may not be an issue anymore with fliter2D

Next Step:

• Maybe shift the image up -480 pixels (but need to check with other images to see if this can be used repeatedly in different situations)
• Try resizing the resulting image after the kernel is applied
• May have to run a manual convolution to see if Kernel is running properly
• Change kernel size to something smaller

[MohammadKhan-3]

Tried resizing the image after the convolution to back to its original size which resulted in the heat map in the correct location but still flipped about the x-axis.

I then tried the cv2.flip() function about both the x & y axis which pushes the heat map to the correct location.

Next Steps:

• Resizing worked, but image is still flipped
• Try resizing Kernel to smaller size. Change from 51x51 to smaller

[MohammadKhan-3]

Changes:

Changed kernel to 5x5

• No cv2.flip()
• No Translation
• No Resize of heatmap back to original 480x620
• Using 5x5 matrix of RGB Blue: [0,0,255]

  Resulting Image:

  

Potential Reason for Previous errors:

• Kernel size might be too large (originally was 51x51) so when the kernel is applied the 51x51 matrix was too large for the image and outputted a smaller dimension image which could have messed with the how the blue square appeared to be detected.
• This site has a great demonstration of how image kernels & convolutions work & this article also has an animation to explain kernels.

Next Steps:

• Overly heat map with camera frame output
• Obtain pixel location of detected blue square from heat map

[MohammadKhan-3]

Contour Detection of heatmap

• using color filter code from color_finder.py --> isolated RGB channels, created mask

• Tried to overlay mask on top of image pulled from realsense camera but not working neatly
• Could try messing around with the masks more to get clean overlay - Purpose: To see how well the kernel matches over the actual image

Obtain pixel locations of blue region

Plan of Action: (In progress as of 11/8) - Find centers of contours Current Error: ~~`raise Exception(("Contours tuple must have length 2 or 3, " Exception: Contours tuple must have length 2 or 3, otherwise OpenCV changed their cv2.findContours return signature yet again. Refer to Open CV's documentation in that case'~~ - need a way to filter out other blue, red, green detected from the actual square

Rechecking Heat Map: 11/9

• at the moment, the filter2D function did not output a gradient of blue values, it is either [255,255,255] (white) or [0,255,255] (Cyan) nothing in between
• Need to check if using filter2D function properly

[MohammadKhan-3]

Rechecking HeatMap

• Used cv2.applyColorMap() on the frame to create a heatmap from that to get a gradient of value

• The image on the left is the filter2D output while the image on the right is the colorMap output

• The colorMap provides a range of values while the filte2D just gives [255,255,255] (white) or [0,255,255] (Cyan)

• Below is applying Kernel (5x5 [0,0,255]) to the color map

[MohammadKhan-3]

Rechecking Heatmap

• Tried changing kernel size from 5x5 to 25z25, 15x15 but it didn't work on creating a gradient. It created the same mirroring effect as before.
• Size of the blue square is 17 x 17 px

Next Steps:

• Review how Kernels work & convolution math
• Look into issues with OpenCV filter2D function to see if there are bugs

[acbuynak]

Found the first issue. I think the kernel was being assembled incorrectly.

Original errored kernel creation

kernel_size = 25
kernel_blue = np.zeros((kernel_size,kernel_size))  # creating an empty array
kernel_blue = np.append(kernel_blue,[0,0,255])
kernel_blue = np.asanyarray(kernel_blue,np.float32)

This creates a float32 array of shape {628,1} containing values of either 0 or 255. Not the desired kernel.

Suggested correction: Note, I'm using uint8 data type to agree with the imported image.

# Create kernel (format - "BGR")
kernel_size  = 5
kernel_b     = 255 * np.ones((kernel_size, kernel_size, 1), dtype='uint8')
kernel_gr    = np.zeros((kernel_size, kernel_size, 2), dtype='uint8')
kernel       = np.dstack((kernel_b,kernel_gr))

Still working on the cv2.filter2D image parsing.

[acbuynak]

Taking some notes here from experimenting with cv2.filter2D

Test Image

Shape: 400x400 pixel. 8-bit.	RGB Image Stack	Grayscale Slices ( R - G - B )

Setup

The cv2.filter2D filter accepts depth arrays, but the source and kernel frames must depth (as expected). Thus, an m x n x d source image must have a kernel of equal depth o x p x d.

Using ImageJ / Fiji to inspect produced images.

Effects of Kernel Size

Kernel sizes tested as ratio of image coverage. ex: 50x50 px kernel on 400x400 px image is (2500/160000)=1.56% coverage Documentation on pixel size (link)

The function uses the DFT-based algorithm in case of sufficiently large kernels (~11 x 11 or larger) and the direct algorithm for small kernels.

Settings: ddepth=-1	1.56% = 50x50	0.39% = 25x25	0.14% = 15x15	0.09% = 12x12

Effects of ddepth Setting

Documentation Link Kernel Size = 15x15	-1 (Match Source Depth)	0	1	2	3
			x	x

[acbuynak]

Alternative Technique using cv2.matchTemplate

Method Options

Documentation Link Kernel Size: 15 x 15 px Usage:

image = cv2.imread("test_image_RGB.tiff")
image = image[:, :, 0]    # cv2 uses BGR format. Thus, blue is channel 0
kernel_size = 15
kernel = 255 * np.ones((kernel_size, kernel_size), dtype='uint8')
res = cv2.matchTemplate(image=image, templ=kernel, method=0)

Option	Filter Result	Option	Filter Result
0		1
2		3
4		5

[acbuynak]

Out of curiosity and the potential for progress using cv2.matchTemplate, I plugged a sample image into each method of the filter to obtain the below results.

Test Image

Original Image is RGB Size: 640 x 360 px

Only the BLUE channel is passed into the below filter.

Testing

image = cv2.imread("sample_image.png")
image = image[:, :, 0]    # cv2 uses BGR format. Thus blue is channel 0
kernel_size = 15
kernel = 255 * np.ones((kernel_size, kernel_size), dtype='uint8')
res = cv2.matchTemplate(image=image, templ=kernel, method=0)

Option	Filter Result	Option	Filter Result
0		1
2		3
4		5	Did not capture

!! Important: Results from Methods 0 and 2 are misleading as to the magnitude of each cell's values. Their grayscale brightness is scaled by ImageJ for readability, but their real value is in the range of 10^-7 or approx zero.

[acbuynak]

In regards to cv2.matchTemplate, method 1 seems to produce the most useful results with a clear gradient across color regions of interest.

[MohammadKhan-3]

Next Steps:

• Create 10x10 px test image of single color (one with a block of a single, one with a angle color, one that is half black and half white)
• Apply 3x3 kernel
• Save output as tiff
• Use ImageJ to read pixel values directly --> check to see if the gradient of values is present

If the above works continue to large image sizes

[acbuynak]

Added convolution testing sample images and test script @ 46aec0aae91f4c821586a45990c2ef51da2f3087

[MohammadKhan-3]

Created 10x10 px image & ran 3x3 kernel using Filter2D

• This was performed in-between classes so I'll redo this to make sure I get the same result.

Input Image: (originally .tiff file but GitHub doesn't support it)

Output Image Array from filter2D: (originally .tiff file but GitHub doesn't support it)

I used Scott Lab computers for this and was not able to check pixel values to see if a gradient was created. Below is the code to create a kernel with 3 channels

Kernel Creation

# Create kernel (format - "BGR")  
kernel_size = 3
kernel = np.dstack((255 * np.ones((kernel_size, kernel_size, 1), dtype='uint8'),np.zeros((kernel_size, kernel_size, 3), dtype='uint8'))) # format: BGR
 # numpy dstack docs: https://numpy.org/doc/stable/reference/generated/numpy.dstack.html
 kernel_b = 255 * np.ones((kernel_size, kernel_size), dtype='uint8')

Filter2D method:

dst = image.copy()
output = cv2.filter2D(src=image, dst=dst, ddepth=-1, kernel=kernel_b)
cv2.imshow('Heatmap',output)
cv2.imshow('Output Array dst', dst)
cv2.imwrite("dst.tiff", dst)
cv2.imwrite("result.tff", output)

Next Steps:

• This post is mainly a placeholder so I know what happened before
• Need to check the pixel values to see if the convolution worked properly

[MohammadKhan-3]

MatchTemplate Function

Notes on how function works:

• Function is extremely sensitive to rotation, viewing angle, and scale
• The result matrix will have a large value (closer to 1), where there is more likely to be a template match.
• Similarly, the result matrix will have a small value (closer to 0), where matches are less likely.

  Results:

  

• Gradient was created but the values in the pixel reader do not seem to match where the blue square is. It's flipped. Higher values are where the blue square isn't' and low where the blue square is

 # # Method: Template Matching
image = image[:, :, 0]
# img_gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
# resb = cv2.matchTemplate(image=image[:, :, 0], templ=kernel_b, method=1)
# resg = cv2.matchTemplate(image=image[:, :, 1], templ=kernel_b, method=1)
# resr = cv2.matchTemplate(image=image[:, :, 2], templ=kernel_b, method=1)
# cv2.imwrite('resb.tiff',resb)
# cv2.imwrite('resg.tiff',resg)
# cv2.imwrite('resr.tiff',resr)
# template = kernel_b[1,:,:]

res = cv2.matchTemplate(image=image,templ=kernel_b,method=1)
cv2.imwrite('res.tiff',res)

• breaking it into the RGB channels yields the same results.

Drew a bounding box on where the function thinks the square is:

Next Steps:

• Continue debugging why the square is detected in the wrong location
• read up on MatchTemplate function in opencv

[MohammadKhan-3]

Checking Filter2D():

• Doesn't out a gradient with the 10x10 px image. Just makes the white square larger
• Input kernel is 3x3x1
• output array: 3x3x3 RGB
• similar results using angle and split images
• increasing image size to 20x20 yields same results

  Code for Filter2D function

  kernel_b=cv2.flip(kernel_b,-1)
  output = cv2.filter2D(src=image, ddepth=-1, kernel=kernel_b)
  cv2.imwrite('output_filter2D.tif', output)

[MohammadKhan-3]

Checking MatchTemplate():

• Generates a gradient
• Also drew a bounding box of 1px width at where it thinks the detected square is
• MatchTemplate extremely sensitive to rotation, viewing angle, and scale
• Note: choosing method 3 works (cv::TM_CCORR_NORMED)

Detecting Box

Detecting Split

Detecting Angle color

---

Conclusion:

As expected, the MatchTemplate is sensitive to subtle changes of the object. It works well if the object is in the same orientation and size but if it isn't, the detection breaks down quickly.

Code for MatchTemplate()

  img_gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
  res = cv2.matchTemplate(image=img_gray,templ=kernel_b,method=3)
  cv2.imwrite('res_match_template.tiff',res)
  min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
  # print values from above

  # Drawing Bounding Box around detected shape
  # determine the starting and ending (x, y)-coordinates of the bounding box
  # From: https://www.pyimagesearch.com/2021/03/22/opencv-template-matching-cv2-matchtemplate/
  (startX, startY) = max_loc
  endX = startX + kernel_b.shape[1]
  endY = startY + kernel_b.shape[0]

  # draw the bounding box on the image
  b_box_image = cv2.rectangle(image, (startX, startY), (endX, endY), (0, 255, 0), 1)

[MohammadKhan-3]

Testing MatchTemplate() with Pixelated Diamonds

• MatchTemplate is sensitive to orientation of the object
• Test to see if the white space can be recognized by the function n

  Observations:

• MatchTemplate is detecting the white square but testing on 10x10 px images isn't representative to testing it on larger images Next Steps
• Test it on Tic Tac Toe images

[MohammadKhan-3]

Testing MatchTemplate() with Tic-Tac-Toe

e

[MohammadKhan-3]

Testing MatchTemplate() with Tic-Tac-Toe Angled orientation

• It did not correctly recognize the blue square
• Expected because MatchTemplate() is sensitive to orientation
• The wrong object was recognized because it fit the closest one to blue

Blue Square Crop:

Code for both Straight & Angled Orientation:

# Reading image
image = cv2.imread("tic_tac_toe_images/twistCorrectedColoredSquares_Color.tiff")
# For straight orientation use: image = cv2.imread("tic_tac_toe_images/CorrectedColoredSquares_Color.tiff")

# creating blue square crop as kernel
kernel_b = cv2.imread('tic_tac_toe_images/blue_square_crop.tiff')

res = cv2.matchTemplate(image=image,templ=kernel_b,method=3)
cv2.imwrite('res_match_template.tiff',res)

cv::TemplateMatchModes cv::TM_SQDIFF = 0, cv::TM_SQDIFF_NORMED = 1, cv::TM_CCORR = 2, cv::TM_CCORR_NORMED = 3, cv::TM_CCOEFF = 4, cv::TM_CCOEFF_NORMED = 5

[MohammadKhan-3]

Using different methods of MatchTemplate() with Tic-Tac-Toe Angled & Straight Orientation

• Using method 5, CCOEFF_NORMED works for both angled & Straight orientation

Looking more into the methods offered by MatchTemplate()

• Here are some examples of MatchTemplate MatchTemplate Docs
• These docs show the parameters for MatchTemplate()
• From what I can gather, it looks like the methods have a varying degree of sensitivity and both method 4 & 5 work in the straight and angled orientations

Next Steps:

• These results were obtained using OpenCV 4.5.3 in Scott Labs. In AIMS, we use 4.2, so need to test on that version of opencv
• Need to test with a live feed
• The Max & Min loc returned by the function are the top right & bottom left corners of the bounding box respectively --> use these to get the center of the bounding box and get orientation of the entire board

[MohammadKhan-3]

Match template: Recognizing all 3 Squares

• Using OpenCV 4.5.3 & method 5 of matchTemplate, you can recognize the squares
• This is a quick proof of concept that you can detect all 3 squares using MatchTemplate()
• The heatmap is for the Green box only

Interesting to note is that if you do multiple Match Template functions in the same script, multiple bounding boxes can be drawn on one image. However, the heat map shown below is the last one in the code (in this case, the green box heatmap)

Code:

# Read Image into script
image = cv2.imread("tic_tac_toe_images/twistCorrectedColoredSquares_Color.tiff")

# Creating Kernels from cropped images
kernel_b = cv2.imread('tic_tac_toe_images/blue_square_crop.tiff')    # blue square detection
kernel_r = cv2.imread('tic_tac_toe_images/red_square_crop.tiff')      # red square detection
kernel_g = cv2.imread('tic_tac_toe_images/green_square_crop.tiff') # green square detection

# MatchTemplate()
res_B = cv2.matchTemplate(image=image,templ=kernel_b,method=5)
cv2.imwrite('res_match_template_B.tiff',res_B)
min_val_B, max_val_B, min_loc_B, max_loc_B = cv2.minMaxLoc(res_B)
print('min_loc_B')
print(min_loc_B)
print('max_loc_B')
print(max_loc_B)

# Drawing Bounding Box around detected shape
# determine the starting and ending (x, y)-coordinates of the bounding box
# From: https://www.pyimagesearch.com/2021/03/22/opencv-template-matching-cv2-matchtemplate/
(startX_B, startY_B) = max_loc_B
endX_B = startX_B + kernel_b.shape[1]
endY_B = startY_B + kernel_b.shape[0]

# draw the bounding box on the image (same process for green & red boxes)
b_box_image = cv2.rectangle(image, (startX_B, startY_B), (endX_B, endY_B), (255, 0, 0), 4) # BGR for openCV
# show the output image
# cv2.imshow("Output based on matchTemplate", b_box_image)
cv2.imwrite('res_match_template_Blue_BoundingBox.tiff', b_box_image)

Next Steps:

• These results were obtained using OpenCV 4.5.3 in Scott Labs. In AIMS, we use 4.2, so need to test on that version of opencv
• Need to test with a live feed
• The Max & Min loc returned by the function are the top right & bottom left corners of the bounding box respectively --> use these to get the center of the bounding box and get orientation of the entire board
• Still need to figure out how to account for rotations

[acbuynak]

we use 4.2, so need to test on that version of opencv

Is there any reason we can't use the newer version of opencv? My understanding of the OpenCV version choice was arbitrary.

[MohammadKhan-3]

Making MatchTemplate more applicable to general scenarios:

• Currently, I used the square as the template and ran that through the image to detect the location of the squares. This is only applicable if the situation uses squares for detection.
• To make the function applicable to general situations, I need to test it using an RGB array to detect color or suggest a different method

Next Steps:

• Make the kernel an RGB array to detect color and the location of the highest true blue color using matchTemplate()
• If MatchTemplate() doesn't work, try other convolution methods like filter2D

[MohammadKhan-3]

Notes on MatchTemplate():

• Input image must match the template image. So if input image is RGB while template image is Grayscale, matchTemplate() won't work
• When using a 50x50 RGB image as the input image and a 3x3x3 RGB array of (0,0,255) an error appears.

  
  ImageJ 1.53a; Java 1.8.0_112 [64-bit]; Windows 10 10.0; 40MB of 24483MB (<1%)

java.lang.NegativeArraySizeException at ij.io.ImageReader.readCompressed32bitImage(ImageReader.java:271) at ij.io.ImageReader.read32bitImage(ImageReader.java:203) at ij.io.ImageReader.readPixels(ImageReader.java:788) at ij.io.FileOpener.readPixels(FileOpener.java:541) at ij.io.FileOpener.open(FileOpener.java:96) at ij.io.FileOpener.openImage(FileOpener.java:53) at ij.io.Opener.openTiff2(Opener.java:1039) at ij.io.Opener.openTiff(Opener.java:842) at ij.io.Opener.openImage(Opener.java:317) at ij.io.Opener.openImage(Opener.java:243) at ij.io.Opener.open(Opener.java:109) at ij.io.Opener.open(Opener.java:72) at ij.plugin.Commands.run(Commands.java:27) at ij.IJ.runPlugIn(IJ.java:204) at ij.Executer.runCommand(Executer.java:150) at ij.Executer.run(Executer.java:68) at java.lang.Thread.run(Thread.java:745)

- I think the error lies in creating a 3 depth array to use as a kernel. Here is the current method of making the 3 depth array:

Create kernel (format - "BGR")

kernel_size = 5 print('Shape of Input image') print(np.shape(image))

Uncomment below to create blue array

ch1 = 255*np.ones((kernel_size, kernel_size), dtype='uint8') ch2 = np.zeros((kernel_size, kernel_size), dtype='uint8') kernel_b = np.array([ch1, ch2, ch2], ndmin=3, dtype='uint8')

might be BGR

print('Kernel Matrix: should be 3x3x3') print(np.shape(kernel_b)) # returns 3x3x3 print(kernel_b)

**Output:**

Shape of image (50, 50, 3) Kernel Matrix: should be 5x5x3 (3, 5, 5) [[[255 255 255 255 255] [255 255 255 255 255] [255 255 255 255 255] [255 255 255 255 255] [255 255 255 255 255]]

[[ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0]]

[[ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0] [ 0 0 0 0 0]]]

## Next Steps: 
- Look into Numpy Array docs for creating a 3D Array
- https://www.kite.com/python/answers/how-to-create-a-3d-numpy-array-in-python
- https://numpy.org/doc/stable/reference/generated/numpy.array.html
- 

[MohammadKhan-3]

Looking into Numpy Arrays & Shape function

Numpy Array: https://numpy.org/doc/stable/reference/generated/numpy.array.html

• The 3D arrays are being made properly, so maybe the issue lies in the difference in the imread()
• Usually, the template is an image loaded into the cv2.imread() function but for general applications, I'm using an array rather than reading an image.
• Need to check how the imread() function works to make sure it outputs an array.

Numpy Shape: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.shape.html

• np.shape(array) outputs a tuple value of (depth, rows, columns)

  import numpy as np
  kernel_size = 5
  ch1 = 255*np.ones((kernel_size, kernel_size), dtype='uint8')
  ch2 = np.zeros((kernel_size, kernel_size), dtype='uint8')
  a = np.array([ch1,ch1,ch2])
  print(a)
  print(a.shape)
  print(a.ndim)

  Output:

  [[[255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]]
  
  [[255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]
  [255 255 255 255 255]]
  
  [[  0   0   0   0   0]
  [  0   0   0   0   0]
  [  0   0   0   0   0]
  [  0   0   0   0   0]
  [  0   0   0   0   0]]]
  (3, 5, 5)
  3

  Next Steps

• a.ndim: Prints out the correct dimension of the matrix (3)
• Need to look into matchTemplate() if the 3D array is messing it up.
• Does matchTemplate() use a 3D array or does it convert it to 1d array? Similar to using the grayscale to look through the image?

[MohammadKhan-3]

Looking more into cv2.matchtemplate():

openCV docs: https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html

• The docs read the image into the template. imread() docs
• From imread() docs: In the case of color images, the decoded images will have the channels stored in B G R order.
• By default number of pixels must be less than 2^30. Limit can be set using system variable OPENCV_IO_MAX_IMAGE_PIXELS

Process for using matchTemplate():

• Read image using cv2.imread()
• take image into matchtemplate() function
• then matctemplate() outputs the heatmap

  img_gray = cv.cvtColor(img_rgb, cv.COLOR_BGR2GRAY)
  template = cv.imread('mario_coin.png',0)
  res = cv.matchTemplate(img_gray,template,cv.TM_CCOEFF_NORMED)

  Next Steps:

• The openCV example uses a grayscale input image and a grayscale template image which makes sense. You need the input image and template image to have the same dimensions (1D for grayscale, 3D for RGB)
• Maybe use a 1D grayscale matrix for (255,0,0) (BGR format) and a grayscale input image
• Grayscale = 0.299R + 0.587G + 0.114B

If matchTemplate() doesn't work: look into other color detection methods

• fliter2D might work but it mainly modifies images rather than detect objects/color
• might need to use HSV extract to detect the color and a different function to detect its location

[MohammadKhan-3]

Plan for completing Tic-Tac-Toe Project:

• Use MatchTemplate() with the image of the blue, red, and green squares as templates
• Obtain the centers for each of the squares found -> draw the axis of the board
• Test using the realsense camera and see how accurate the orientation detection is
• Finish this first before creating a general use of matchTemplate()

Creating General Use of matchTemplate()

• General Use: using a true blue, red, green RGB values as the template to search for in the image.
• The difference between this method and the one used in tic-tac-toe (TTT) is the use of the template images. In TTT, I use the image of the colored square. After that, the same method to obtain the board orientation is used. (Find centers of the detected squares, draw the x & y axis, obtain angle of orientation using slope)

[MohammadKhan-3]

Live Camera feed with MatchTemplate() function

• MatchTemplate Docs: https://docs.opencv.org/4.2.0/de/da9/tutorial_template_matching.html

• Uses older version of Python (2.x) with Raspberry Pi but inputs camera feed without converting file types: https://stackoverflow.com/questions/42559985/python-opencv-template-matching-using-the-live-camera-feed-frame-as-input The above link imports from picamera.array import PiRGBArray from picamera import PiCamera This is a Raspberry Pi specific import. See here for more info: https://picamera.readthedocs.io/en/release-1.13/

• This example is in C++ and imports a video into matchTemplate() not a camera feed. Not exactly what I need but the discussion of drift in video tracking is interesting: https://stackoverflow.com/questions/20180073/real-time-template-matching-opencv-c

Recognizing objects at different scales:

• Interesting topic on how to recognize the same object at different distances from the camera. https://stackoverflow.com/questions/43994713/python-opencv-live-camera-feed-scale-object
• They use template matching & loop over different resolutions of the input image to recognize it at different scales.

More importantly, the use a live camera feed:

#Camera 
cap = cv2.VideoCapture(0)

#symbool inladen
symbool = cv2.imread('klaver.jpg',0)
w, h = symbool.shape[::-1]

while(1):
    res, frame = cap.read()

In my case, I am subscribing to a rostopic. image_sub = rospy.Subscriber("/camera/color/image_raw", Image, runner) However, I could forgo subscribing to the topic and access the camera feed directly since the script does not need to pull from the image topic. The purpose of the script is to recognize where the squares are and output the centers and orientation so the robot can make the appropriate movements to play the game.

Action Items:

• Try accessing the video camera feed directly without pulling from the rostopic. However there were issues in the past with using cv2.VideoCapture(0)

[acbuynak]

However, I could forgo subscribing to the topic and access the camera feed directly since the script does not need to pull from the image topic.

Heads up, I don't think it's possible to have both a ROS-based camera feed AND a python-based feed open at the same time. I don't remember ever testing this specifically, but I assume due to the handshake between the camera and computer that it might not let you initiate the second stream.

[MohammadKhan-3]

However, I could forgo subscribing to the topic and access the camera feed directly since the script does not need to pull from the image topic.

Heads up, I don't think it's possible to have both a ROS-based camera feed AND a python-based feed open at the same time. I don't remember ever testing this specifically, but I assume due to the handshake between the camera and computer that it might not let you initiate the second stream.

Ok. I'll see if it can be done. Just to clarify I was going to remove the ros topic and ros imports and use just cv2.VideoCapture() for accessing the camera feed.

[acbuynak]

Just to clarify I was going to remove the ros topic and ros imports and use just cv2.VideoCapture() for accessing the camera feed.

Gotcha. No worries, just wanted to help catch anything before we got there.

[MohammadKhan-3]

Potential Solution to cv2.VideoCapture() issues

• I know we've had issues with cv2.VideoCapture() before with using cv2.VideoCapture(0) or cv2.VideoCapture(1), etc

This Stack Overflow post helped solve it: https://stackoverflow.com/questions/52029233/how-to-make-usb-camera-work-with-opencv

TLDR: Plug in the camera, go to your terminal home

• Type cd /dev
• Type ls video and then press tab, if you find only result as video0, that means only webcam is present.
• Now repeat 1 to 2 with USB webcam plugged in. You should find video1 or video2 when you repeat the steps.

When I plugged in iRS#2 camera and cd /dev with the above steps, I had video0, video1, video2, video3, video4, video5 appear. I found that cv2.VideoCapture(4) worked.

[MohammadKhan-3]

Live Video feed using cv2.Video Capture: Image is dark

• cv2.VideoCapture worked in opening the camera however the image appears dark which disrupts the blue square detection

Tried adding a time.sleep() but even at time.sleep(30), the image remains dark. Image is shown below

Here is the code I am using:

        cap = cv2.VideoCapture(4)
        # refer to this github issue for why I used VideoCapture(4)
        # https://github.com/OSU-AIMS/tic-tac-toe/issues/10#issuecomment-1016505927

        # allowing the camera time to boot up and auto set exposure
        time.sleep(30) # seconds

        while(1):
            res,frame = cap.read()
            frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
            kernel_runner(frame)
        cap.release()

Opening the camera in Intel Realsense viewer is fine. The video feed brightens up within 1 sec.

Action Items:

• Look into waiting some number of frames (~100 frames) rather than time to see if that provides enough time for the camera to fully set the auto exposure.
• If the above works, test to see if the blue square detection still works.
• If detection works, test red & green then output the center coordinates to a different script to obtain the board's orientation.

[MohammadKhan-3]

MatchTemplate: needs to be more robust

• Currently using method used for static images applied to a dynamic environment.

• As seen above, it does recognize the blue square with a live-feed but any rotation outside the input image (kernel) will result in an incorrect detection.

To make MatchTemplate() more robust detection will need to look into keypoint detectors, local invariant descriptors and keypoint matching to detect changes in scale, rotation, lighting, etc in the image.

Note:

• The static image had the board at a different distance away from the camera. Using the kernels for the static image will not work in the above dynamic environment because the distance from the board to the camera is different which results in the colored squares being different sizes in the camera feed.
• MatchTemplate() is sensitive to rotation, scaling, lighting, camera focal length. What works in one setting, may not work in another if the environment are different.

Next Steps:

• Look into Keypoint matching & other methods to making MatchTemplate() more robust

Potential Local Invariant Descriptors & Keypoint Matching methods to look into: SIFT, SURF, FREAK, RANSAC

• SIFT: https://www.analyticsvidhya.com/blog/2019/10/detailed-guide-powerful-sift-technique-image-matching-python/
• SURF:https://medium.com/data-breach/introduction-to-surf-speeded-up-robust-features-c7396d6e7c4e
• KNIFT: https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html
• FREAK:https://ieeexplore.ieee.org/document/6247715
• RANSAC:http://www.cs.cmu.edu/~16385/s17/Slides/10.3_2D_Alignment__RANSAC.pdf

@acbuynak if you've heard of any these before or any suggestions on which to start with, let me know.

[acbuynak]

Responses

Using the kernels for the static image will not work in the above dynamic environment because the distance from the board to the camera is different which results in the colored squares being different sizes in the camera feed.

Understood. let's assume a sufficiently static environment for now where the board will always be a fixed z-distance from the camera. Before we jump onto making this more adjustable, let's run the techniques past some vision experts to get ideas/guidance first. See last note.

MatchTemplate() is sensitive to rotation, scaling, lighting, camera focal length. What works in one setting, may not work in another if the environment are different.

Makes sense, our first draft implementation is going to be rough.

@acbuynak if you've heard of any these before or any suggestions on which to start with, let me know.

Nope. These are all new to me, but let's pause on robustness if it's working even slightly at this point. We could run them past mgroeber to see if he recognizes them later.

---

Next Steps

Let's hold off for now. I'd rather have you help Luis to get the rest of the structure setup. Even if the vision is only working sometimes.. that's okay. We just want a basic prototype before proceeding.

[MohammadKhan-3]

Final results:

Using MatchTemplate(), able to recognize 3 colors with a decent accuracy. Not perfect --> lighting, rotation, and other dynamic conditions still affect detection

Issues:

• Light above robot is harsh and not uniform so there are slight shadows that mess up detection
• MatchTemplate() sensitive to changes in environment --> needs to be more robust

For more info on Object detection Methods and making MatchTemplate() more robust. See the Object Detection Wiki Page in the right panel: https://github.com/OSU-AIMS/tic-tac-toe/wiki