search

stephanlensky / hyacinth

A Discord bot to send notifications for marketplace (Craigslist, Facebook) postings based on complex matching rules.
https://slensky.com/hyacinth
GNU Affero General Public License v3.0
45 stars 8 forks source link

Question about rate limiting #71

Closed d0xi1337 closed 6 months ago

d0xi1337 commented 7 months ago

Hey, apologies if this area is not where this type of question belongs, i don't use github but i've been searching for this type of software and came across this. I have some questions:

  1. When does rate limiting typically occur from your testing? Could you give an estimate like 30 searches every 10 minutes
  2. Does the program find new listings outside your local area? If so would it be possible to only search for listings with shipping enabled?

Thanks

stephanlensky commented 6 months ago

Hi, this is a fine place to ask!

When does rate limiting typically occur from your testing? Could you give an estimate like 30 searches every 10 minutes

Unfortunately it's difficult to give a definitive answer here. It depends on a few different factors, including:

  1. When the searches actually execute. Currently, searches which poll on the same interval will execute all at once, making a ton of requests very quickly and potentially causing rate limiting. Ideally, I should stagger the requests a bit so that they are evenly spread out over the long (usually around 10 minutes) poll interval. I have an open issue for that https://github.com/stephanlensky/hyacinth/issues/21, but haven't gotten around to it yet.
  2. The plugin you're using to search. Craigslist and FB Marketplace will have different rate-limiting rules.
  3. How many new listings are found each time the search is polled (every new listing = 1 network request)

To reduce the number of searches you need, I would suggest using the most general category possible for the actual search, and then filtering down the results you want using filter rules. For example, if you were interested in a specific type of car (e.g. Honda Civic) and a specific type of motorcycle (e.g. Yamaha R6), rather than creating two searches for the FB Marketplace cars and motorcycles categories, you could create a single search on vehicles and then filter for Honda Civic and Yamaha R6.

If you use that same vehicles search in multiple channels (e.g. #cars and #motorcycles), Hyacinth will be smart enough to combine the polling, thus reducing the number of requests you're making.

Generally, I would suggest sticking to just a few searches in order to avoid getting rate-limited or banned.

Does the program find new listings outside your local area? If so would it be possible to only search for listings with shipping enabled?

(I assume you're asking about FB Marketplace) when you configure the search, you need to enter the location to search in (see docs). Separate areas would need to be separate searches. As far as I can tell this is a limitation of Marketplace that can't be worked around.

It should be possible to filter for listings with shipping enabled, but I haven't implemented this yet. Happy to accept a contribution if you are interested in adding this.