rndme / download

file downloading using client-side javascript
MIT License
2.28k stars 417 forks source link

anyone had problems using many chrome parallel downloads? #113

Closed nmatei closed 1 year ago

nmatei commented 1 year ago

I'm using this library to download files from our API. But now i see that if I start more than 80 requests instantly and then I download them using this lib, some files are not downloaded (no errors seen in console or in requests). Requests seems to be there with status: 200, but there are missing files in my downloads folder. This works ok in FF. Anyone had this issue? I've managed to improve this by not starting all requests when user press 'download' all, I've created a Queue and when some requests are done, i'm starting others.

nmatei commented 1 year ago

if delay is 0 not all files are downloaded, it works only if we delay them.

var delay = 0;
new Array(100).fill().forEach((o, i) => {
  const timeout = Math.min(i*delay, 60000)
  setTimeout(() => {
    console.info('start', timeout);
    // download(
    //   //`https://cdnjs.cloudflare.com/ajax/libs/downloadjs/1.4.8/download.js?i=${i}&_dc=${new Date().getTime()}`
    //   //`https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js?i=${i}&_dc=${new Date().getTime()}`
    //   `https://code.jquery.com/jquery-3.6.1.js?i=${i}&_dc=${new Date().getTime()}`
    // )
    download("hello world", `dlText-${i}.txt`, "text/plain");
  }, timeout)
})
rndme commented 1 year ago

I've noticed the same thing before. There's a CPU spike when a download (of any size) starts, likely from updating logs, filesystem, Chrome's Downloads UI and it's DB, etc. I'd imagine that Chrome throttles or discards downloads in an "extreme" situation like that to prevent freezing, which people would blame on chrome. I'm not sure of the heuristics or hard limits in play, but a queue or small timeout would be a good way around it. It would be hard to fix generically inside of download.js as that would presumably entail buffering an unknown number of files of unknown length on all user agents, some of which are limited in ram, like phones. I'll keep it in mind, and see if I can come up with a way that doesn't overly complicate the implementation. If I were you, I would continue using your queue strategy, or consider zipping them up and downloading one zip, using something like https://gildas-lormeau.github.io/zip.js/ .