search

benbusby / farside

A smart redirecting gateway for various frontend services
https://farside.link
MIT License
776 stars 47 forks source link

Use searx.space to see if SearXNG instances are online #95

Closed return42 closed 1 year ago

return42 commented 1 year ago

The implementation of farside instances do a search request on each SearXNG instance to see if the instance is online and does it work.

From the view of a SearXNG instance, the sum of the farside instance behave like a bot net that has been gone wild / doing the same query:

https://github.com/benbusby/farside/blob/6048d3c23ea5bcc289747989f305c38a5b4b9ecb/config/config.exs#L16-L18

.. again and again, 24/7 from different IPs. In SearXNG we tend to block wild bots since they will have a negative impact on the engines of the SearXNG instance when we pass trough these requests.

My suggestion:

To avoid this flood of requests on each SearXNG instance, the farside instances can use the list from the SearXNG team to see what instances are online --> https://searx.space/

May the farside team has more / other suggestions to improve the situation / Thank you for your understanding.

benbusby commented 1 year ago

Understandable. I'll just remove the periodic polling for SearXNG and update the nightly build to factor in instance availability when generating the instance list.

unixfox commented 1 year ago

We can also increase the frequency of checking the health of each instance. @dalf proposed to do that, I'm not what's the current frequency (one per hour)?