search

lonekorean / wordpress-export-to-markdown

Converts a WordPress export XML file into Markdown files.
MIT License
1.14k stars 227 forks source link

Image download delay #32

Closed derekphilipau closed 4 years ago

derekphilipau commented 4 years ago

In writer.js, function writeImageFilesPromise(posts, config), line 81, the delay increments by 100ms. For large WP sites on slow/overburdened servers, this can result in what amounts to a DoS attack, with exceptions (RequestError: Error: read ECONNRESET) proliferating as the server becomes overloaded.

After modifying the delay to increment by 1000ms instead of 100ms, I successfully downloaded each file, although of course it was slow. Luckily most users won't need to run this software frequently.

Suggest allowing user to specify image download delay or at least raise the default delay to something higher, perhaps 500ms.

Otherwise software worked a charm.

lonekorean commented 4 years ago

Hey thanks, yeah that's something I've wrestled with. Want it to be fast... but not too fast. :)

Long term I want to update the code to not exceed X pending requests, but for now I'll just increase the delay.

To give me a better idea, how many images did you have to download?

lonekorean commented 4 years ago

Increased the default delay to 500ms. Also moved those variables into a settings file.

https://github.com/lonekorean/wordpress-export-to-markdown/commit/d8c513ff04d6610f73d15950fd2e6a6b3e89945b

Thanks again!