search

kakaroto / R20Exporter

Script to export Roll 20 campaign into a zip file.
54 stars 14 forks source link

downloadResource() gets overwhelmed #17

Closed kyleady closed 2 years ago

kyleady commented 4 years ago

R20Exporter sends out a flurry of requests to Roll20 on each step. If the campaign is large enough (as in my case) then network errors will start to pile up. The function downloadResource() will give up after the first network error, leading to a large number of missing blobs in the final ZIP file.

I solved this issue by adding in an exponential backoff to downloadResource(). However, this exponential backoff does not occur when receiving 404 Not Found or 403 Forbidden. I tested this out on an old campaign that had a large number of images that were added through the Roll20 search and Roll20 marketplace. The lack of retires when being hit with 403/404 kept the time to ZIP unaffected.

My modifications for adding the exponential backoff to v0.7 can be found here.

kakaroto commented 3 years ago

Why am I just seeing this now! 😭 I don't know why, but I didn't have this repo watched for some reason. Thank you very much for the exponential backoff algo you added. I'll be merging that in soon!

kakaroto commented 2 years ago

This has finally been merged and released in 0.8.0. Thanks again and sorry for the delay!