search

talleyhoe / google-image-scraper

Simple google images scraper without chromium
GNU General Public License v3.0
19 stars 6 forks source link

Cannot retrieve images #3

Open thecoderenroute opened 2 months ago

thecoderenroute commented 2 months ago

Since 4/17/24, the script is not functioning and shows the following in the terminal:

image

script was working as of 4/16/24 and zero changes have been made since.

I'd appreciate any help in this matter.

talleyhoe commented 2 months ago

Looks like google has blocked the endpoint. Asking for 'asearch=ichunklite' returns a status 400 error.

I'll have to figure out something else to make this work. For now, sadly, I'd recommend you use another project.

Thanks for letting me know!

thecoderenroute commented 2 months ago

Thanks for the response! I'm trying out the official Google API and see if I can make it work within the rate limit. However, I'd request you to keep the issue open so I can work on it later if that's okay.

talleyhoe commented 2 months ago

Definitely going to leave this open until the project is functional again. Finding undocumented exposed google endpoints isn't an easy task. If you find anything please lmk, I'd love the help. I have some other hacky ideas for their javascript, but tbh web dev is definitely a weak point for me so I'll probably procrastinate.

Looks like google allows 100 results per day for free. Utilizing this could be a good alternative in the meantime. Some (uninformed) thoughts on design: scripting the key generation -> user configs (this part sucks) utilizing the api to generate a json manifest (url and data modifications) downloading via that manifest (mostly drop in to existing code)

Sounds like you can get by without the scripting portion of this, which will probably take the most time. What's your timeline? I haven't messed with the api, but we might be able to get you a patch in a day or two.

If you start working on something, open up a pull request (I might open my own if I don't see yours)! I'd be happy to work on this with you!

albertusgeyser commented 1 month ago

I used your code to scrape some images for a online shop that is still in dev. Just had some time to implement a api call to your code then i realize it broken. This a sad one as i looked for code like this for a while an yours were only one i could find working months back and now it broken due to google api changes as mentioned.

I really hope you could make this to work again.

Thank you for sharing code.

talleyhoe commented 4 weeks ago

Hey, I've thought about this for a while. I think the ethos of this project should centered on unfettered access to public images. So I've decided to create another scraper that utilizes the official API as a stop gap for now. Feel free to follow along with the development, I expect to be done in a few weeks. I'll update the readme and another comment here when it's ready for use.

https://github.com/talleyhoe/google-image-scraper-gapi

Fixing this repo/issue will take me a while, unless someone (community or a friend) steps in and helps me understand/guides some of these javascript complexities. Tbh, probably won't have time to seriously look at this until Jan 25.

I appreciate all the love, honestly didn't think anyone would use this :)

talleyhoe commented 4 weeks ago

Scratch that. The google API limits are abysmal. I've found a source that I should be able to reverse engineer to formulate a proper workaround. Still going to be a couple weeks, but stay posted.

Happy Hacking :)