Street View API - Handling downloads at scale

Streets-Data-Collaborative / groundwork-detection

Use customvision.ai to detect specific elements of urban infrastructure.

2 stars 5 forks source link

Street View API - Handling downloads at scale #6

Closed karansaini282 closed 5 years ago

karansaini282 commented 5 years ago

The Street View script written by @karansaini282 crashes at large scale of downloads (~8000 downloads).

@sunghoonyang has suggested that a good way of scaling this script is to run it in smaller batches and keep moving the downloaded images to maybe a Google Drive.

vr00n commented 5 years ago

OK. Can you share the script on this issue thread please? Please remember you are limited to 5,000 images in CustomVision free version.

sunghoonyang commented 5 years ago

@karansaini282 hi karan, I ran into HTTP 403 error after 6000 images... Is this the same error? It's not system crashing so I wouldn't think so. I'm writing images directly to mounted google drive fs, so I think im evading the problem u had. I will see if try/catch-ing 403 error can help me run full iteration of Manhattan road sweep..

vr00n commented 5 years ago

@karansaini282 @sunghoonyang - can this be closed? what was the resolution?

sunghoonyang commented 5 years ago

The new ETL code with direct write to mounted Google Drive file sys allowed ETL to finish without errors. I produced 75k images during 3 days without any failed execution