Determine research direction

[SebastianZimmeck]

It turned out to be difficult to use the VPN for different countries. Here are a few options to try:

• Ask @katehausladen if she has some insight (@danielgoldelman)
• Different VPN (e.g., Nord VPN) (@dadak-dom, @natelevinson10, and @JoeChampeau)
• Different countries/regions (@dadak-dom, @natelevinson10, and @JoeChampeau)
• Google Cloud (@natelevinson10)
• Investigating network layer to check what is up (WireShark etc.) (@danielgoldelman)

[katehausladen]

Ask @katehausladen if she has some insight

I have only used the Mullvad VPN in Los Angeles for crawling. I haven't ever used it for other countries. My only suggestion would be trying multiple different IP addresses in the same location if the VPN has more than one.

[SebastianZimmeck]

Thanks, @katehausladen!

Indeed, it seems like the GPC crawl is different than the Privacy Pioneer crawl. @danielgoldelman will describe a bit more the problem to you and maybe you can think of additional ideas. This is just in case.

[JoeChampeau]

As discussed during the meeting, trying out anyIP's static residential proxies didn't bear any fruit. Seems like proxies in general may not be able to offer any advantages over cloud computing or VPNs, although it might still be worth trying out what @natelevinson10 found.

[dadak-dom]

I've been running test crawls on the Google Cloud VM. As @danielgoldelman mentioned, the cloud option provides a significant improvement for location elements, but performs worse for the analytics category. As discussed in the meeting, I'll start looking into what could possibly be going wrong here.

[natelevinson10]

Similarly to @JoeChampeau , I also tested static ISP proxies from the provider Bright Proxies and found that they did not offer and advantages either. With these findings, we can likely close the door on the possibility of using web scraping proxies to test the crawler.

[SebastianZimmeck]

Sounds good, @natelevinson10!