Open gasolin opened 2 years ago
My workaround is to automtically change IP whenever blocked. I'm using windscribe VPN, following the instructions in the first part this article. Once manually logged in through terminal, I'm able to change IP by just one Windscribe command without any input prompted, which makes it easy to integrate into Python code.
Some pseudo-code snippets:
os.system("windscribe connect US")
try:
get_info(url)
except TypeError: # get_info(url) returns None when reached maximum request limit from same IP
os.system("windscribe connect US")
get_info(url)
今天試發現 豆瓣限制一個 IP 短時間內一次最多抓10頁,
稍微改了一下加入
pagination = 1
參數如下調整 pagination 值, 搭配不同 VPN server 可以全抓下來