lonicade
commented
3 years ago Might want to try slowing down your refresh rate... some site admins set thresholds, and block when you refresh too much. Also, you could try scrambling/chunking your list of urls in your .yaml, if you're using a combined one (verses by domain), so that you aren't doing all the pulls at one site back-to-back. It might be valuable to have an argument to randomize the scrapes a bit, to make it look less structured/bot-like..
Hello, I tried to set up this bot for some shops in my area (Germany). For some vendors, it says "access denied" (see log). Also, i put in a link with a 3090 that I know is in stock just for testing, but the bot says it is not. When I run the container, it gives me warnings that it's only using generic scrapers.
Does this bot only work with US vendors? What do I need to do to make it work in my region? Or am I just missing a configuration step?
Thanks guys!
[LOG]
W2021-01-11 21:18:05,484 [root] warning: using generic scraper for url: https://www.computeruniverse.net/de/pny-geforce-rtx3080-xlr8-gaming-epic-x-rgb-m-10-gb-enthusiast-grafikkarte I2021-01-11 21:18:05,485 [cmptrnvrs_nt_32] scraper initialized for https://www.computeruniverse.net/de/pny-geforce-rtx3080-xlr8-gaming-epic-x-rgb-m-10-gb-enthusiast-grafikkarte ... I2021-01-11 21:24:10,540 [ltrnt_d_1] not in stock I2021-01-11 21:24:12,588 [ltrnt_d_2] not in stock I2021-01-11 21:24:14,555 [ltrnt_d_3] not in stock I2021-01-11 21:24:17,413 [ltrnt_d_4] not in stock I2021-01-11 21:24:19,400 [ltrnt_d_5] not in stock I2021-01-11 21:24:21,141 [cskng_d_1] not in stock I2021-01-11 21:24:23,381 [cskng_d_2] not in stock I2021-01-11 21:24:25,687 [cskng_d_3] not in stock I2021-01-11 21:24:27,841 [cskng_d_4] not in stock W2021-01-11 21:20:29,411 [cmptrnvrs_nt_1] access denied, got HTTP status code 403 (forbidden) W2021-01-11 21:20:31,593 [cmptrnvrs_nt_2] access denied, got HTTP status code 403 (forbidden) W2021-01-11 21:20:33,859 [cmptrnvrs_nt_3] access denied, got HTTP status code 403 (forbidden)