Harris Corner Detection

[matthewb96]

The Harris corner detection works at finding the corners for the test image line and outputting an image showing these corners, shown in this picture, Harris corner detection was run using a function in open CV. However this does not give the positions of the corners in order to find this out you must use the connectedComponentsWithStats() function to find the centroids, this finds the average position for each corner. This can then be run through another function cornerSubPix() to find a more accurate position of the corner. This process is not getting the desired results because the function to find the centroids is not working correctly. It only find 3 centroid positions and these are not at the positions of the corners. A binary image was created using threshold() in open CV that then could be analyzed as an array instead of using the open CV functions, the binary image was created in this commit. #https://github.com/matthewb96/FibreLengthAnalysis/commit/5f56efd984c52c32a7e552123fcd45f408a5d25c In order to solve this problem there are a few ideas that have been thought of by me and my supervisors during our meeting (30/10/17):

• Plot histograms of the greyscale images after each function to see how the functions affect the arrays
• Possibly normalize the images after each function if the greyscale show differences, the functions could alter the greyscale image enough to make it difficult for the next function to work
• Could try using a centroid finding function in scikit image on the numpy arrays instead of the open CV functions
• Also there could be an issue with the Harris corner detection function altering the original image so when the original image is ran through the cornerSubPix() function it does not work correctly. Therefore running the Harris corner detection on an image copy might be better.

[matthewb96]

First thing to try was to create histograms after each function to have a better understanding of how the functions change the greyscale. Relevant commit #https://github.com/matthewb96/FibreLengthAnalysis/commit/1353cf463e9a419df8b760f7231721c288e4c361 Here are some images with differing amounts of bins. Default bin number, (i think 10). Much larger number of bins to see a more precise position, some bins are too small to see. The first two histograms are mostly at about 250 which is expected because they are mostly white images with some black, hence the smaller peak at near 0. The Corners histogram has a bigger peak at 0 because the corners algorithm colours the pixels so that the more white pixels are at a corner, so there are much more black pixels than white. Threshold turns the pixels either black or white so the peak at 0 is for the same reason as above but not completely sure why it goees from -0.5 to 0.5 because all of the arrays have been converted to 8bit intergers (between 0 and 255).

Edit: Added link to relevant commit

[matthewb96]

After plotting the histograms for each step I have printed the amount of data points in the original image histogram in order to check the correct amount of data is being plotted, this is data for a 100*100 pixel image.

This shows 30000 data points in the original image, 3 for each pixel because it is an RGB JPEG image. For the grayscale image there is 10000 data points because the grayscale image only has one number per pixel. This shows that the histogram has the number of data points expected so the histogram is not missing any data. Commit: #https://github.com/matthewb96/FibreLengthAnalysis/commit/66e00a447f459c43e7a89b58cfe36d3d104f13af Edit: Added the commit this refers to.

[matthewb96]

Saving the image produced by Harris corner detection function, as well as coloring the positions found by connectedComponentsWithStats() and cornerSubPix() onto the original image. The image used has only one corner and is a 100*100 pixels JPEG. Here is the original image: Here is the Harris Corner detected image, showing just the position of the corner in white: Here is the centroids and sub pixel corner positions as found by the above functions: The Harris corner detection image looks to have plotted the corner in the correct position and has only plotted one corner, so it looks as though cornerHarris() is working exactly as is needed. However both the centroids and sub pixel positions are not at the correct position and also there are 3 sets of coordinates when only one is expected, the exact coordinates of those positions are below, red being for the centroids connectedComponentsWithStats() and blue being for the sub pixel coordinates cornerSubPix(). This data is from the same run of the program as the data in the comment above, and is also from the same commit, #https://github.com/matthewb96/FibreLengthAnalysis/commit/66e00a447f459c43e7a89b58cfe36d3d104f13af

[matthewb96]

Trying 3 different images to test how the system works with each tested a triangle, a square and a simple fibre like image, shown below. The triangle and square image have been stretched because they are smaller images so blurring is seen. Here are the Harris corner images for each test. Here are the centroid and sub pixel positions. All three images show the corners in the correct place from cornerHarris(), although the triangle is not very clear. However none of the images show the centroid or sub pixel positions in the correct location. Although they do all seem to be in the center and all have 3 positions for each, this could show that the centroids are an average of all the corners but that doesn't explain why there are 3 centroid positions for each image. Also because these centroid positions are so far away from the corners this could mean cornerSubPix() cannot find any nearby corners so by default is just returning the same position as the centroids. Therefore the only function that is deffinately not working as intened is connectedComponentsWithStats(), cornerSubPix() may or may not be working correctly. Another function to find the centroids may be needed. Here are the coordinates for the centroid positions (red) and the sub pixel positions (blue) for each image respectively. The coordinates for centroids and sub pixel positions differ by ~10^(-6) for each separate image, considering the positions of the pixels are integers that shows that these coordinates are the same. The slight difference could be due to values being converted between types or some other small rounding error. Commit: #https://github.com/matthewb96/FibreLengthAnalysis/commit/8e809c484e9c63b0d269ccc464447e264b4dd3b3

[matthewb96]

Debugging the part of the code to find the centroids as it was not working as expected. A break was set so that the lines leading up to the connectedComponentsWithStats() could be ran one at a time and the variables could be looked at after each step. cornersThres was created as an ndarray from the previous corners array then threshold() was used to create a binary image. However threshold() returns a tuple containing the threshold value and the ndarray, so when connectedComponentsWithStats() was used it was being given a tuple and throwing an error. The code was changed so that the correct data was saved to cornersThres variable and this allowed connectedComponentsWithStats() to work. This returned a lot of different centroid positions, some of which seem to accurate. However there is only one corner in the image so only one centroid was expected, editing how the thresholding is being done may help this. Commit: #https://github.com/matthewb96/FibreLengthAnalysis/commit/2365908d119d60a497aa8ce1867b7bcf578cbbbf

[matthewb96]

The top 3 images are of the same ndarray (cornerThres) at separate points in the program. This first image is when the cornerThres variable is created as an 8-bit version of the original corners array created by cornerHarris() then edited by dilate(). This second image is after threshold() has been ran on the array with a threshold value of 240, i.e. everything above 240 is set to 255 and everything below is set to 0 to create a binary image. Third image is same as above just showing a different part of the array. This image is of the centroids array created by connectedCornersWithStats(). Each of these positions corresponds to 255 in the array, apart from row 3 this seems to have found the average position of the four values around it. Row 3 is the position that is wanted. Positions 2 and 4 correspond to values of 243 on the image before thresholding so these can be avoided simply be increasing the threshold value to 250. However positions 1 and 5 have the same value as the points surrounding position 3, increasing the size of the array that connectedComponentsWithStats() averages would find only one set of coordinates as the corner point, which is what is wanted. Unsure as to why a centroid position is at position 1 as the value here is 0, if this position is consistent could be an error in the function that may just need to be accounted for by checking if any of the surrounding values are not 0. Commit with threshold value changed to 250 (https://github.com/matthewb96/FibreLengthAnalysis/commit/163315881eeebf41ff598977a85c33ae3937c60d), which shows 4 centroid positions as expected. Having connectedComponentsWithStats() average a larger array would get this down the the one corner position wanted, as well as the row 0 error.

[matthewb96]

Commit, https://github.com/matthewb96/FibreLengthAnalysis/commit/163315881eeebf41ff598977a85c33ae3937c60d , was measuring the centroids properly but when looking at the centroid positions on the original image it is clear they are not at the corner. Harris Corner Detected Image Centroid image Previously assumed that the larger end of the Harris corner detected image was at the corner but it appears to be the thinner end at the corner so the centroids found previously are not in the correct position. This could be due to the use of the image edited by dilate() trying running the original cornerHarris() image through connectedComponentsWithStats() and see if that gives a better result.

This was done in commit, https://github.com/matthewb96/FibreLengthAnalysis/commit/74afd4cd0d04f1afe913688d0b6aa732eb95ae86 , but showed an almost identical result. Harris corner detected image (without dilate()) Centroid Image To see the difference between the images produced by cornerHarris() and dilate() here is the array unedited by dilate(). This is very similar to the image in the previous comment (first image), so this does not help get a more accurate picture of where the corner is. The actual position of the corner is circled above this seems to be the center position average of all the surrounding values, so this could be found by averaging a bigger array. The threshold value needs to be reduced to include all surrounding values, a good value to try would be 150, then an array size of about 25*25 should encompass all the values. Below is the grayscale image showing the exact position of the corner.

[matthewb96]

https://github.com/matthewb96/FibreLengthAnalysis/commit/82e53e907912278b818e2395ecaf3020a7577810 Reverted back to using dilate() as this gives a larger number of points round the edge which should make finding the center of all the points more accurate and easier. Corner array edited with dilate() Corner array unedited with dilate() The next step is to change connectedComponentsWithStats() so that it will average over a much larger array of 25*25 and therefore find the correct corner position and only one position per corner.

[matthewb96]

Looking at how the variables change when using cornerHarris() in order to see if there is something wrong there, when looking at the array in variable explorer the values are larger around the corner position, larger = more blue, this image shows the array after cornerHarris() but before dilate() and also before in has been converted to 8 bit. Another break point was added after the threshold to see the cornerThres array, threshold() was done on the corners array after editing by dilate() but without converting to 8 bit image, this is the array returned, with the corner position found above drawn on. This group of pixels is close to the exact corner. When trying to run the next line it threw a value exception so the cornersThres array is converted to 8 bit before connectedComponentsWithStats() is run. This produced centroids with 2 positions one the average position of the 255 values and the other is most likely the average of all the 0 values, which makes sense as it finds average positions of anything connected. After debugging it was ran again, not in debug mode, to see the results and it had found the corners perfectly. Corner positions red from connectedComponentsWithStats() and blue from cornerSubPix() cornerSubPix() gives an accurate position of the corners and should just give a list of corner positions with the first position being the average of all 0 values which is not needed. Commit, https://github.com/matthewb96/FibreLengthAnalysis/commit/3d4e1454a50352990e75b9928ce02eeeedd185b1

[matthewb96]

Tested with the fibre like image and has found the corners to within 1 pixel accuracy zoomed in images showing exact corner positions on original image. Top of fibre Bottom of Fibre The red pixel is the centroid, found using connectedComponentsWithStats() and the green pixel is the sub pixel positiom, found with cornerSubPix(). Here are the coordinates, again the 0 value average is the first set of positions cornersSubPix[0] this consistency suggests it is always the first set so these can be ignored. Although should be checked on more samples. Commit https://github.com/matthewb96/FibreLengthAnalysis/commit/eec4d2b34399cfa96e23f61f4e4e2495e6b68f7d

[matthewb96]

The corner positions have been found using Harris corner detection as the above comment shows so now the midpoint along the short edges need to be found so the length of the fibres can be calculated.