koito19960406 / ZenSVI

This package is a one-stop solution for downloading, cleaning, analyzing street view imagery
https://zensvi.readthedocs.io/en/latest/
Creative Commons Attribution Share Alike 4.0 International
25 stars 5 forks source link

Use proxies and User agent for kartaview downloader #87

Open koito19960406 opened 2 weeks ago

koito19960406 commented 2 weeks ago

I get the following error after downloading a few hundred images:

Error: HTTPSConnectionPool(host='storage13.openstreetcam.org', port=443): Read timed out

Using proxies and user agents might solve this issue.

koito19960406 commented 6 days ago

KVDownloader already has user agents and proxies. It might be due to other reasons.

def worker(row, output_dir, cropped):
            url, panoid = row.url, row.id
            user_agent = random.choice(self.user_agents)
            proxy = random.choice(self.proxies)

            image_name = f"{panoid}.png"  # Use id for file name
            image_path = output_dir / image_name
            try:
                response = requests.get(
                    url, headers=user_agent, proxies=proxy, timeout=10
                )
                if response.status_code == 200:
                    with open(image_path, "wb") as f:
                        f.write(response.content)

                    if cropped:
                        img = Image.open(image_path)
                        w, h = img.size
                        img_cropped = img.crop((0, 0, w, h // 2))
                        img_cropped.save(image_path)

                else:
                    if self.logger is not None:
                        self.logger.log_failed_pids(panoid)
            except Exception as e:
                if self.logger is not None:
                    self.logger.log_failed_pids(panoid)
                print(f"Error: {e}")