search

jywarren / irkit

Raspberry Pi-based infrared/visible image compositing and webcam control (Public Lab)
2 stars 10 forks source link

finding corresponding points in each image #3

Open jywarren opened 11 years ago

jywarren commented 11 years ago

[ax1, ay1], [bx1, by1], [ax2, ay2], [bx2, by2] etc

dwblair commented 11 years ago

This might be overkill (Jeff might have a much simpler, faster method in mind for finding corresponding points in each image), but: I'm following the (really nice!) tutorial in this document:

http://programmingcomputervision.com/downloads/Draft_20120318.pdf

which uses the Python Imaging Library (along with scipy and numpy) to identify common features in two images. Specifically, I'm relying on the "Harris Corner Detector" section, described starting on p. 45, and I've implemented the code (verbatim) up to p. 52, where this algorithm is used to compare two images. Here's the result (using two webcam images taken of a wall mural):

wavematches

It looks (sort of) promising! Potential problems, though:

One idea might be to run a function like this once, on startup, using some printed pattern of grids in front of the two cameras (or taking to snapshots of images at "infinity", if that's better), in order to 'calibrate' the matching, and determine the proper transform -- then, in the air, only the transform would need to be done. Another option would be to simply save the matching for later, using an online interface ...

Whee!

jywarren commented 11 years ago

very cool! what sized image did you run it on? i like the idea to run it only when you "hold the button down" - then re-use that relationship -- it should hold as long as the subject is more than a few meters away

Also: run it on even more low-res images (this is probably the best thing to try first) -- like 320x240 or something

Also, you could run it for only 20 or so matches rather than all the ones you have above. But probably it takes about as long to find interest points as it does to match them. But it could save a bit.

I'm assuming you'd have to look at the transform each matched pair indicates, then discard outliers, then we could use that as part of an affine transform. Looking good!

On Sun, Feb 10, 2013 at 7:48 PM, dwblair notifications@github.com wrote:

This might be overkill (Jeff might have a much simpler, faster method in mind for finding corresponding points in each image), but: I'm following the (really nice!) tutorial in this document:

http://programmingcomputervision.com/downloads/Draft_20120318.pdf

which uses the Python Imaging Library (along with scipy and numpy) to identify common features in two images. Specifically, I'm relying on the "Harris Corner Detector" section, described starting on p. 45, and I've implemented the code (verbatim) up to p. 52, where this algorithm is used to compare two images. Here's the result (using two webcam images taken of a wall mural):

[image: wavematches]https://f.cloud.github.com/assets/639760/144016/eaa8901e-73e3-11e2-8299-4809a40d111f.png

It looks (sort of) promising! Potential problems, though:

-

It's really slow. For now, I'm doing this on my laptop, and it took around 2 minutes to find these common features. I haven't dug into the algorithm, though -- there might be some tricks to speed things up;

It might not be the most robust algorithm. The following section in the above document employs the (patented?!) SIFT algorithm, which is supposed to be pretty much the state-of-the-art. Not sure re: relative speed.

One idea might be to run a function like this once, on startup, using some printed pattern of grids in front of the two cameras (or taking to snapshots of images at "infinity", if that's better), in order to 'calibrate' the matching, and determine the proper transform -- then, in the air, only the transform would need to be done. Another option would be to simply save the matching for later, using an online interface ...

Whee!

— Reply to this email directly or view it on GitHubhttps://github.com/jywarren/irkit/issues/3#issuecomment-13365103..

dwblair commented 11 years ago

great ideas!

jywarren commented 11 years ago

wow, sounds great. I think we can tweak thing a lot once we get it running, but keeping a file structure like this could help keep things under control:

/photos/infrared/ /photos/visible/ /photos/infrared-aligned/ (assume we only adjust the infrared images, not the visible ones?) /photos/nrg-composites/ /photos/ndvi-composites/

We can come up with more user-friendly names later, i guess.

On Tue, Feb 12, 2013 at 9:05 AM, dwblair notifications@github.com wrote:

great ideas!

  • this was a 640x480 image, and the process would be faster if it had been lower-res;
  • there was an easy option in the code for finding only the first N matches; will try that too;
  • good point re: outliers -- not sure off the top of my head how to decide what are 'good' vs 'bad matches'; i suppose that we compute an average predicted 'translation' of features in a particular region of the image, and use that to identify 'outliers', according to some cutoff ... i.e. if our algorithm indicates that most pixels in a region are 'off' by around 4 pixels, but that one of them is 'off' by 100 pixels, then we discard the feature that suggested that latter transform ...

  • yes, it sounds like a nice way forward would be to calibrate the two webcams initially on the ground (with some sort of printed-out test pattern, say); or, as Chris F. had suggested, post-flight. Chris pointed out that he sometimes likes to fiddle with the process of finding proper overlaps in his images. If it turns out that matching imagines mid-flight is too processor intensive, I suppose it's not terrible to do things this way? I.e. you need to 'develop' your film via some interface ... this script (on your laptop, or e.g. online at publiclaboratory.org) could be more or less no-frills to begin with, but could ultimately allow the user to tweak the relative brightness, alignment, and contrast of each image, say ...

    — Reply to this email directly or view it on GitHubhttps://github.com/jywarren/irkit/issues/3#issuecomment-13433612.

dwblair commented 11 years ago

Update:

I'm still trying to wrap my head around this, but: there is a very simple interface to the very popular "OpenCV" platform for computer vision called "SimpleCV", and it's apparently possible to install on an R-Pi -- though it might be easier using ArchLinux.

I installed SimpleCV quite easily on Ubuntu, and -- woah -- there are so many fun functions to play with, now! 3D reconstruction ... depth maps ... face recognition ... oh my!

But, back to the task at hand. I tried to match key features in the same images I'd used in an earlier test, using the "drawKeypointMatches" function.

match

Here's the entire code necessary to do the matching:

from SimpleCV import Image template = Image('wave1.jpg') img = Image('wave2.jpg') match = img.drawKeypointMatches(template,490.00,0.05) match.save("match.jpg") match.show()

It was relatively much faster than the code used above (due to its C++ libraries, I suppose) -- it only took about 1 second on my 2 ghz core duo laptop. Will try installing it on the R-Pi and see how fast it runs.

If this ends up being a possible way forward, having SimpleCV / OpenCV at our disposal opens up a lot of neat possibilities -- including, I believe, 3D reconstruction / depth maps ... maybe once we get the basic code up and running, we can attract the attention of some computer vision geeks who really know their way around these sorts of applications ...

dwblair commented 11 years ago

So -- as soon as I figure out what's going wrong with my latest install of the official Wheezy install (2013-02-09-wheezy-raspbian.zip) (something seems to be going wrong at the very first "sudo apt-get update", to do with 'signature verification' -- I'll dig in and see what's wrong -- might even consider switching to Arch Linux (there might be benefits to doing this re: computer vision packages, anyway) ... THEN the task at hand will be to figure out which of the following approaches (perhaps a blend of them both) works best in achieveing some easy to use computer vision libraries (esp. SimpleCV) on the Pi:

and / or

One nice thing about that latter link: they've already set up the proper code for accessing the GPIO pins -- meaning, button and buzzer action soon to follow!