benbusby / whoogle-search

A self-hosted, ad-free, privacy-respecting metasearch engine
https://pypi.org/project/whoogle-search/
MIT License
9.57k stars 944 forks source link

[BUG] Instance has been ratelimited #707

Open eddydc opened 2 years ago

eddydc commented 2 years ago

Describe the bug Getting

Error

Instance has been ratelimited

Continue your search with Farside Whoogle: https://farside.link/whoogle/search?q=dit is een test&lang_interface=lang_en&country=BE&theme=system

Searx: https://farside.link/searx/search?q=dit is een test

Return Home

To Reproduce Just do a search with Whoogle

Deployment Method

Version of Whoogle Search lastest

Desktop (please complete the following information): Win 10, latest FF

T3chHunt3r commented 2 years ago

I too have this issue on version 0.7.0

update: This issue seems to be resolved for me, I re-composed the docker container as well as changed locations for my VPN. I'm not sure which of the two fixed the issue however. It's more likely that it was a VPN issue as I was also blocked from certain sites due to the IP that I was under at the time.

ShlomiD83 commented 2 years ago

same issue for me. can't resolve the issue no matter what I do.

sysoppl commented 2 years ago

Same for me, just deployed and it's not working at all.

ironboy1 commented 2 years ago

Same on my side for version 0.7.1 and 0.7.2

bruvv commented 2 years ago

This has been answered already: https://github.com/benbusby/whoogle-search/issues/211

pieslinger commented 2 years ago

I am having rate limiting issues occasionally and I am the only user of whoogle running in a container. I do not see how #211 has answered this already. I would install this for more people in my life as well if this issue goes away. Is there just something I need to modify in the setup?

pieslinger commented 2 years ago

I also only noticed it in 0.7.1 and 0.7.2 - although 0.7 was my first time using this. Just installed 0.7.3 tonight.

update: I use Pihole as well

aw-jansen commented 2 years ago

Same here, currently on 0.7.3. Getting the rate limiting message when connected via a NordVPN container. Would it still be a possibility to implement a workaround? On a different topic someone mentioned changing the query: https://github.com/searxng/searxng/issues/159

aindriu80 commented 2 years ago

I'm having the same error, only making a few requests and get this error, I'm using Pi-Hole and wonder has that anything to do with it?

whaler-ragweed commented 2 years ago

Same here on 0.7.4 using a public IP with password. Getting limited for average one or two days. Restarting the docker can fix it, but the problem always appears again.

jojolll commented 2 years ago

I use my own Whoogle instance (alone). Almost every day I reach the ratelimit after 10 to 15 searches during the day.

I have the chance with my LiveBox to be able with a simple click in the interface to renew my IP, that solves the problem for 1-2 days. It's still very annoying and makes the solution of a personal Whoogle server very unviable.

Albonycal commented 2 years ago

there isn't much you can do about this issue, google blocks the requests it thinks are suspicious. If your IP is dynamic, try rebooting or change your VPS's IP.

dr460nf1r3 commented 2 years ago

This might in fact also be caused by bots crawling the instances. At least I noticed that the rate limits became less after blocking most known bots via Cloudflare WAF (12k events over 24h) 🤔

CptanPanic commented 1 year ago

I am having this issue today also. I am not sure what #211 has to do with this, is there any debug to look at to see what is causing the issue?

bruvv commented 1 year ago

For me. A restart of the docket container fixes this.

whaler-ragweed commented 1 year ago

Same here on 0.7.4 using a public IP with password. Getting limited for average one or two days. Restarting the docker can fix it, but the problem always appears again.

I moved my VPS form Linode to Amazon Lightsail 5 months ago and never have the issue again.

ForceConstant commented 1 year ago

It is not clear from the error message, but looking at the code, the ratelimit is external to whoogle, and it is coming from a service that whoogle is querying. And if it gets a captcha, then it reports this. So I couldn't imaging how @bruvv restarting your container would affect this.

0xspade commented 1 year ago

For me. A restart of the docket container fixes this.

Doesn't work on my side.

CptanPanic commented 1 year ago

For me, it turns out it was the proxy I was using. I guess there was too many people at that IP address.

On Wed, Jan 11, 2023 at 4:45 AM 0xspade @.***> wrote:

For me. A restart of the docket container fixes this.

Doesn't work on my side.

— Reply to this email directly, view it on GitHub https://github.com/benbusby/whoogle-search/issues/707#issuecomment-1378484128, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7F3ATE3JW43U4YK52W263WRZ6MTANCNFSM5SHZHBZQ . You are receiving this because you commented.Message ID: @.***>

Paulie420 commented 1 year ago

So... I get this with both whoogle-search and googler - only when connected to my Private Internet Access VPN on some of their locations. Canada being one of them.... I'd bet a lot of folks seeing this issue are simply behind a VPN that Google is blocking - if this describes your setup, try disconnecting from your VPN and running whoogle-search again; viola, fixed.

Snake16547 commented 1 year ago

Same issue here after 2 months of whoogle usage. Tried to restart docker container - still rate limit exceeded

So basically moving to different machine would solve the problem? Any recommendations?

dr460nf1r3 commented 1 year ago

From my understanding it's the IP getting rate-limited, so using another one should work.

Snake16547 commented 1 year ago

Easy to say to change the IP when you have a VPS

brokoler commented 1 year ago

Since the last 2 days all of the public Whoogle instances are affected by this, most times only one of them works and yesterday it happened that even with Farside all instances were rate limited.

To me it seems like Google is going against Whoogle, but I still hope it's just a bug in the latest Whoogle software that can be fixed.

@benbusby do you have a clue what is happening right now?

Snake16547 commented 1 year ago

I switched to the Oracle free tier and until now I'm free of any ratelimit messages.

benbusby commented 1 year ago

@brokoler public instances go through waves of all being rate limited, and then being clear again. It looks like some are available now. There's no simple fix to get around it unfortunately, but it isn't a bug in Whoogle. I always recommend using your own Whoogle instance, or running it locally on your machine (behind a VPN if you want) if rate limiting becomes too much of a problem.

MaverickTN commented 1 year ago

I just had this issue for the first time after updating to 0.8.2. I last used it 3 days ago, and it's rate-limited on all queries. Restarted the container, no change. Private instance, closed network, docker container.

Johnnii360 commented 1 year ago

I'm joining this issue. Same problem on my instance. Maybe it could be a workaround to use an API key in the developer console but it has to be implemented by @benbusby . It's a common thing that Google shows an CAPTCHA request if there too many search request of one and the same IP. I think it's their "DDoS protection".

Edit 1: Another solution could be the usage of Tor service. But there must be an function that every new visitor is bind to a new Tor connection. If Whoogle needs also one and the same Tor connection it will also end in this issue. Otherwise the Tor service or connection has to be refreshed every day or hour or whatever. I will give it a try.

Edit 2: Okay, bad idea. :D

Edit 3: Uhm... I just changed the country now to my country (German) and also the search language. Now it works again. But the question ist: How long? :)

robertpartridge commented 1 year ago

I've run into this problem today with my deployment of version 0.8.2. My instance is behind basic HTTP auth with a fairly strong unique password so that it's not open to just anyone to use. Any advice?

Edit - reading through more of the responses and wondering if the problem might be related to the instance being deployed on Oracle's Cloud. Will investigate.

seirdotexe commented 1 year ago

Getting this too now on my own IP address, locally hosted. Already restarted the container and even reinstalled it.

akovia commented 1 year ago

Just installed whoogle 0.8.2-1 for the first time ever from Arch aur, so have no frame of reference of how it's supposed to even work.

Is there anything I can do to get this working? I'm behind a VPN, so is that an actual issue to run a self hosted instance?

Deviantgit commented 1 year ago

Have also started running into this--I would get it occasionally over the past year and a half I've been using Whoogle, but generally it'd go away after a few hours. For the past two weeks or so I have only been getting the rate limited message despite taking down my docker container and waiting over 24 hours, etc. This is with it both behind a VPN as well as a non-VPN IP. It's rather odd.

imTHAI commented 1 year ago

Today I face the same problem. But I use it on an internal docker container, and I'm the only one who use it in the family. How is it possible that I have reached the rate limit of search requests for google, from a single home user ?! 🤔

EDIT: Even if I make the whoogle container goes out on internet through the internet connection of my parents (retired people using very little internet, living in a country other than me), I get this rate limit error.

winhex commented 1 year ago

Same problem. I'm deploying the docker container with whoogle 0.8.4 at my own home server. No public access, only personal using. Instance has been rate limited after 20-30 min of using

Drallas commented 1 year ago

I also have been rate limited, my Whoogle runs in a tteck LXC container. Updating the OS doesn't solve it!

zidanpragata commented 12 months ago

whatever time, if direct search method is not changed to the brilliant idea of "yacy search", error message "rate is limited" will still occur frequently.. the easiest solution to overcome this is to change the IP..

failed :)

aindriu80 commented 12 months ago

I tried changing IP and using VPN but still getting rate limited, very frustrating

akovia commented 12 months ago

I bit the bullet and signed up for Kagi now that they made unlimited searches more affordable. I find it better than Google in so many ways and doubt I'll ever go back. I have no skin in the game, but they have a free trial if you want to check it out.

Either way I hope you get it figured out.

Snake16547 commented 12 months ago

I had it running on Oracles free tier with zero problems. Several clients used my instance. Now I switched to Hetzner Cloud and running smoothly as well.

imTHAI commented 12 months ago

Could someone explain why a whoogle instance can be sometimes rate-limited when the other computers (behind the same public ip) aren't when using google ? Are the requests from whoogle so special (and harmful?) that Google identifies them and limits them ?

Drallas commented 12 months ago

I bit the bullet and signed up for Kagi now that they made unlimited searches more affordable. I find it better than Google in so many ways and doubt I'll ever go back. I have no skin in the game, but they have a free trial if you want to check it out.

Either way I hope you get it figured out.

Whoogle is nice, but the search results from Google feel manipulated and without still optimized to show payed results first. Would be better if Whoogle could include more search options. Kagi might be an alternative too!

MuubNito commented 11 months ago

ALL instances i am trying or doesnt work or just straight up give me another issue... Ouch

imTHAI commented 11 months ago

I don't know about you, but I can use my whoogle container again without any problems. Maybe google changed something on his side ?

seirdotexe commented 2 months ago

Since a week, my instance is getting rate limited constantly now, only after 20 searches. Makes it impossible to use Whoogle

C8opmBM commented 2 months ago

happened to me as well 2 times today.

EasternPA commented 1 month ago

I just got this for the first time ever. I am the sole user of a docker-compose stack on Oracle Free Tier. I recently lost SSH access due to rebuilding my PC, so I need to rebuild the cloud instance anyway. Perhaps a rebuild will clear it up.

Zundrium commented 3 weeks ago

Same here, CHAPTA seems to be the issue