Reduce the number number of S3 PutRequests and size of uploads - Githubissues

hotosm / MapCampaigner

Campaign management tool

BSD 3-Clause "New" or "Revised" License

38 stars 23 forks source link

Reduce the number number of S3 PutRequests and size of uploads #860

Open dakotabenjamin opened 4 years ago

dakotabenjamin commented 4 years ago

MapCampaigner doesn't scale too well with numerous large campaigns. It makes a ton of regular PutRequest s3 API calls which can add up over time.

Some initial thoughts to improving this problem:

smart updating based on user interaction
regularly update partial data with manual option for full updates
limit AOIs, campaign lengths, zoom levels, etc.
rather than delete all date and reupload, intelligently update (sync?)