search

hubot-archive / hubot-google-images

A hubot script that interacts with the Google Images API
126 stars 67 forks source link

Google AJAX Image Search: "This API is no longer available" #29

Open seriallos opened 8 years ago

seriallos commented 8 years ago

Looks like Google may have finally killed the API:

$ curl https://ajax.googleapis.com/ajax/services/search/images\?v\=1.0\&q\=testing
{"responseData": null, "responseDetails": "This API is no longer available.", "responseStatus": 403}%
JasonMore commented 8 years ago

Yeah they turned it off. The new api has 1000 req/day limit too :-( :-1:

wjr1985 commented 8 years ago

:cry:

HaroldPutman commented 8 years ago

So maybe we fallback to random cat picture http://lorempixel.com/200/200/cats/? random cat

jeffbyrnes commented 8 years ago

Super sad.

onlydole commented 8 years ago

Are there any other services that could be used instead?

afeld commented 8 years ago

Some options:

darrinholst commented 8 years ago

I tried setting up CSE and failed miserably. Do you really have to specify the domains you want to search? This is a sad, sad day.

seriallos commented 8 years ago

Once you've created the custom search engine with one random domain in it, you can edit and choose "Search the entire web" (or something named like that) in the Basic tab. You can't do it when creating the CSE though.

darrinholst commented 8 years ago

oh nice, any tips on finding the key? I found the id.

seriallos commented 8 years ago

Google API Console. Might have to set up a new project and enable the CSE API.

https://code.google.com/apis/console

darrinholst commented 8 years ago

found it by stumbling around in there. Thinking about starting a side business to set this up for people :dizzy_face:

AngelMarquez commented 8 years ago

Hey guys:

I know it's not perfect, but that's roughly 261 searches per day ... anyone game?

sadatay commented 8 years ago

I started a thread over in the corresponding lita plugin and I've been thinking of doing something similar to @AngelMarquez's suggestion, perhaps also with yahoo support. Basically start with google, fallback to bing, then fallback to yahoo which goes for a rate of $1.80 per 1000 requests (or $0.20 with advertisements). There is also pixabay which provides an unlimited (though rate limited to 2k requests per 30 minutes) image search api - although it's only public domain images. So for example things like "flower" have suitable results but "there's something about mary" has no results. So maybe first hit pixabay to filter out things that might be relatively common before going to the more expensive APIs for broader results.

My coffeescript game is weak so I dunno how much help I can be on the hubot side, but I'm "game" in the sense that I'm game to pursue this idea. It might be smart to put a little thought into the architecture so that we can sort of frame the individual apis as components, so maybe other apis can be easily mixed in or out at will. A generalized "image fetcher" plugin, i guess, with as many sources as people want to implement. There are certainly many possible configuration options when we start talking about an amalgamation of sources like this.

For what it's worth, I also explored scraping a bit today. It definitely violates Google's terms of service, so this wasn't something I planned to release publicly on GitHub, but if anybody is considering handrolling their own image scraping plugin for their chatbots, at least with ruby (for lita) I was not able to find anything suitable. Best I could get were the thumbnails - the whole "click an image to get options" thing means the actual sources of the images aren't really accessible on the page (as far as I can tell). And rigging up some jankiness with, say, phantomjs to go through clicking all the links and getting the sources would be prohibitively slow.

I really want this functionality, I'm sure many can attest that our team chats are less lively without it. I'm pretty sure what you're suggesting here is, for now, the most reasonable course of action.

wjr1985 commented 8 years ago

I like the idea of Google first, falling back to Bing, but I can't seem to find an option to request animated images through Bing's image API. Anyone else have any luck? I tried things like Style:Animated, Style:AnimatedGIF, and Style:GIF set in ImageFilters, but I would get no results.

HaroldPutman commented 8 years ago

The same issue exists for Google CSE (see issue #13). We work around it by setting file type to GIF and adding animated to the query string. Not perfect, but better than nothing.

nzifnab commented 8 years ago

@AngelMarquez I'm trying to follow the README's instructions for setting up a CSE for use w/ HUBOT, but it says I need a CSE Key by going here: https://code.google.com/apis/console

But that URL redirects me to a google developer's console landing page with no clear direction of where to get that key from... I got the CSE ID, but not sure what key I'm supposed to use :/

EDIT: I figured it out, but the instructions could be clearer. You need to generate an API key in that console, and then go to the "Custom search API" link from the console page and "Enable" that API.

AngelMarquez commented 8 years ago

I have modified this script https://github.com/github/hubot-scripts/blob/master/src/scripts/bing-images.coffee a tiny bit and have it working by itself now. When I get some time (hopefully this weekend) i'll try to merge it into this script. The hard part will be cutting over after free searches are exhausted, then knowing when to switch back.

wjr1985 commented 8 years ago

I gave integrating Bing as backup a shot as well, if anyone's interested: https://github.com/hubot-scripts/hubot-google-images/compare/master...wjr1985:bing_fallback

It doesn't stop sending requests to Google when the quota has been exceeded, but it does fall back properly to Bing. I'm sure it could use some cleanup and an updated Readme, but it's working right now in my Hubot integration with Slack

AngelMarquez commented 8 years ago

@wjr1985 I love it!

I think we can just take your code and tweak it by adding a new skipGoogle var. When we have to resort to Bing just set skipGoogle to true and make a call tosetTimeout() to switch skipGoogle back to false in an hour. Not perfect, but it makes it so I'm not hammering Google and speeds up the searches overall.

HaroldPutman commented 8 years ago

@technicalpickles, @louim, @seriallos and other contributors... I'm interested in your thoughts about the Bing fallback. Since this is a "ship-with" command, is there any impetus to keep it simple? Should we spin off a more robust hubot-images command that could draw from a wide range of sources? Is there some more modern "middleware" way to keep google-images, bing-images, and potentially other *-images as separate commands and have hubot "load balance" between a selection of options?

technicalpickles commented 8 years ago

is there any impetus to keep it simple?

Simplicity should be kept in mind for default scripts, for sure.

Should we spin off a more robust hubot-images command that could draw from a wide range of sources?

That makes sense to me.

Is there some more modern "middleware" way to keep google-images, bing-images, and potentially other *-images as separate commands and have hubot "load balance" between a selection of options?

I'm not sure middleware will help here, as it applies to everything. It'd be possible for individual scripts to add a function like robot.googleImageSearch, and let hubot-images or whatever check for those to be present before using them.

shana commented 8 years ago

http://api.imgur.com/ might also be an option. There's a gallery search endpoint (http://api.imgur.com/endpoints/gallery#gallery-search). They have rate limits (12500 a day), and there's the possibility of having our app whitelisted if we ask nice.

JasonMore commented 8 years ago

@wjr1985 I maintain a fork of this code for another project, here is how I got gif working in bing [spoiler alert, its gross]

    if(animated) {
        // arg why?! no good api unfortunately
        qs.Query = `'${query} ".gif"'`;
    }

basically tacking a ".gif" to the end of my query

ex: pikachu ".gif"

afeld commented 8 years ago

Found another option for GIF search: http://api.riffsy.com/

josephks commented 8 years ago

@nzifnab How did you get the CSE Key? I still can't figure it out. I've generated a server key but what field from that JSON get stuck into HUBOT_GOOGLE_CSE_KEY ?

nzifnab commented 8 years ago

@josephks

First, make sure you created a project and generated an API key for it:

https://console.developers.google.com/apis/credentials

That might get you to the right page

After you have your project, you should be able to generate the server key (HUBOT_GOOGLE_CSE_KEY)

next, go here: https://console.developers.google.com/apis/library

You should see a big list of API links, in the search box type "Custom Search",

There should be one result for "Custom Search API", go into it and click "Enable"

I think when you define your custom search engine, it might give you the ID at that time. See this article: https://developers.google.com/custom-search/json-api/v1/overview?hl=en_US for info about creating your custom search engine, it does look like you'll get the ID once you've done that.

EDIT: You can create your custom search engine here: https://cse.google.com/manage/all

Make sure you let it "search the entire web but emphasize included sites" (and just leave included sites blank), and make sure "image search" is turned on.

There's a search engine ID button there that'll give you the ID to use in hubot when you're done.

josephks commented 8 years ago

When I generate the server key I get a json file. Am I supposed to stick all of that json in HUBOT_GOOGLE_CSE_KEY ?

nzifnab commented 8 years ago

er... no? What're you doing to get a JSON file?

nzifnab commented 8 years ago

If you go here: https://console.developers.google.com/apis/credentials

you should create a new project, then generate a "Server Key" and it should give you an API key that's just a string of characters, that's the HUBOT_GOOGLE_CSE_KEY

josephks commented 8 years ago

I see a button that says "Create credentials". After I click on that what do I click on?

nzifnab commented 8 years ago

"API Key", and then "Server Key"

josephks commented 8 years ago

Image search still isn't working :(

nzifnab commented 8 years ago

And you created a custom search engine and used the ID it provided for HUBOT_GOOGLE_CSE_ID?

josephks commented 8 years ago

Yes.

murrant commented 8 years ago

Is there a way to enable Custom Search API without putting in billing information?

moktarul12 commented 8 years ago

try this

moktarul12 commented 8 years ago

free google unofficial image search api. http://api.ababeen.com/ Demo: http://api.ababeen.com/api/demo.php

theapache64 commented 7 years ago

I've developed an API named 'GPix' that can act as a API for the google images. Take a look at https://github.com/theapache64/GPix and if there's any error report at theapache64@gmail.com . It's fresh and hot.!! enjoy! :)

EugeneKovalev commented 7 years ago

@theapache64 OMG That's awesome! Thank you! :)

theapache64 commented 7 years ago

@cyberthrone3 You're most welcome. :)

saggiyogesh commented 7 years ago

@theapache64 Is there any daily quota for search ?

moktarul12 commented 7 years ago

Try api.ababeen.com

On Nov 25, 2016 11:41 AM, "Yogesh Agrawal" notifications@github.com wrote:

@theapache64 https://github.com/theapache64 Is there any daily quota for search ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hubot-scripts/hubot-google-images/issues/29#issuecomment-262887357, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOZW7ZPTXMmIKQ7yzAAqHCV69Ewfw7Kks5rBnv3gaJpZM4GtVbE .

theapache64 commented 7 years ago

@saggiyogesh It's UNLIMITED. All you want to have an api_key. :)

theapache64 commented 7 years ago

I've developed an unofficial API for Google images. It's faster and simple. https://github.com/theapache64/GPix

EugeneKovalev commented 7 years ago

@theapache64 hello again! I'd like to tell you that i get the same new brand API key every day :) hope you can fix this

theapache64 commented 7 years ago

@cyberthrone3 It's not a bug. I changed the algorithm to [one email]-[one-API-key] model. You can use your API key for multiple applications. There's no difference.

theapache64 commented 7 years ago

@seriallos , @JasonMore , @wjr1985 , @HaroldPutman , @jeffbyrnes , @onlydole , @afeld , @darrinholst , @AngelMarquez , @sadatay , @nzifnab , @technicalpickles , @shana , @josephks , @murrant , @moktarul12 , @theapache64 , @cyberthrone3 , @saggiyogesh I've developed an unofficial API for Google images. It's faster and simple. https://github.com/theapache64/GPix . Please take a look and report if there's any issue. It's hot and fresh.. Enjoy. :)