davidfstr / Crystal-Web-Archiver

Downloads websites for long-term archival.
http://dafoster.net/projects/crystal-web-archiver
60 stars 5 forks source link

Allow increasing the default request timeout (of 10 seconds) to download slow sites #198

Open davidfstr opened 5 months ago

davidfstr commented 5 months ago

@pandavova reports in the following message that it would be useful to increase Crystal's default 10 second timeout (per URL) when downloading particularly slow sites. Quoted below:

Are you trying to download a site where the default timeout of 10 seconds per URL isn't long enough?

yeah, really bad host. got it to work after several hours.

@pandavova Two questions:

* Would you still find a _configurable_ timeout to be useful?

* May I know what the slow host is, so that I run some of my own tests on it?

Yes, a configurable timeout would be great! This is the site: https://www.ufopaedia.org/index.php?title=Long_War_Rebalance

Looks like it works right now better then usual, sometimes its a 524 timeout. The page is loading in ~1500ms right now if I understand the firefox metric correctly.

Priority: Low