ArchiveTeam / seesaw-kit

Making a reusable toolkit for writing seesaw scripts
Other
69 stars 30 forks source link

Tumblr rate limits by IP, consider partnering with VPNs to adjust IP periodically #108

Closed timkatje closed 3 years ago

timkatje commented 5 years ago

So, maybe this is a symptom of having really great Internet (Google Fiber 1G), so I can download a lot of content from Tumblr. But I noticed after just a single night of contributing to your project, that there was basically no traffic running through the VM. Later, when trying to just access my own personal blog, I was hit with a Access Denied error message that vaguely mentioned rate limits. If I activate my own VPN service, then I'm able to get past Tumblr's block.

However, your install instructions have us bridge our VMs (VMware) to our physical NICs, so it's more difficult for me to force your traffic to use my VPN, unless I do something at the router level.

As such, for big projects with very short timelines like this one, you may wish to include VPN options inside the VM... either just support many of the most popular ones and give us a way to configure our settings to connect using our accounts. Or, better yet, find a way to partner with a sympathetic VPN company and just do it automatically.

Unfortunately, it's probably too late in the case of Tumblr, but we might run into a similar situation in the future.

Ganonmaster commented 5 years ago

I don't think the use of VPNs is something to be encouraged, as you will likely end up sharing the endpoint with other users. You might actually end up getting the IP address of the VPN endpoint rate limited; potentially making the VPN endpoint unusable for other people using it for legitimate browsing.

You want more, different IP addresses. I'm running the docker image on a set of VPSes, all with different IPs, which is likely a better solution than a VPN.

chfoo commented 5 years ago

Linking a related discussion so it doesn't get lost: ArchiveTeam/warrior-dockerfile#27

TomGlass commented 3 years ago

As discussed at link above, we're heavily against VPNs