search

propublica / facebook-political-ads

Monitoring Facebook Political Ads
MIT License
237 stars 50 forks source link

reverse image search via perceptual hashing #74

Open jeremybmerrill opened 6 years ago

jeremybmerrill commented 6 years ago

it would be really neat to be able to search for ads that use an image (e.g. a news photo or a meme) OR group ads that use the same image.

The way to do this would likely be to -- at some point -- record each image's perceptual hash, then at search time, compare the given image to all images in the DB, returning those that are similar. dhash and phash are both perceptual hashing algorithms, but there may be other good choices.

For instance, hopefully, we could determine which ads use the Distracted Boyfriend meme like this one .

jeremybmerrill commented 6 years ago

any interest, @imalsogreg ?

jeremybmerrill commented 6 years ago

Since my understanding is that these perceptual hash algorithms are primarily architected in Python, this could either happen as part of the classifier classify step in Python in this repo, or in a similar, but new, separate script that's also run on a frequent cron.

imalsogreg commented 6 years ago

That sounds like a lot of fun :) Let me see if I can get a proof of concept up.

jeremybmerrill commented 6 years ago

Awesome, holler if you have Qs or if there's anything I can help with.

yinleon commented 6 years ago

Hi, I just stumbled on this.

I have experience making reverse image search engines using features extracted from pre-trained neural networks, and then calculating distance using KNN. There's some more info (code, presentation, video) in this repo: https://github.com/yinleon/pydata2017

Let me know if this method sounds interesting for this project (sorry for the shameless self-promotion).

jeremybmerrill commented 6 years ago

Hi @yinleon, that sounds amazing. In my response to your other comment, I posted a link to download the entire dataset of US political ads.

This sounds potentially really promising. Are the extracted features interpretable (e.g. "this one has a picture of Nancy Pelosi")? Can you describe how well it works identifying nearly-identical images versus more heavily modified or cropped images? What about images with text overlays, like memes?

I'm not super familiar with Keras. Would it be possible for me or a colleague to test your implementation on an ordinary MacBook Pro? Or is it easier to set it up on a GPU-enabled AWS instance?

I really appreciate your taking the time to share about your research. You're of course welcome to download and check out our data and I'd love if we could find a way to work together. Happy to talk more about my qualitative observations of the ad images, if that'd be helpful.

yinleon commented 6 years ago

@jeremybmerrill I sent you an email to talk in depth about this.

imalsogreg commented 6 years ago

This sounds awesome!

Pre-trained ResNet probably has lots of features that are relevant for general image search (common objects, body parts), and lacks some that would specifically help categorize political ads (Nancy Pelosi neurons, e.g.). I wonder how hard it would be to retrain ResNet with lots of extra labeled examples, with those labels drawn from politically relevant topics. Your google image scraper could be useful there?

I'd always assumed image classifiers would be terrible for image search, but from your video, apparently it can work great, when you don't throw away all features but the strongest one :P Cool result with KNN search.