hgouveia / node-downloader-helper

A simple http file downloader for node.js
MIT License
247 stars 54 forks source link

Downloaded size exceed 100% #87

Closed khkassem closed 1 year ago

khkassem commented 2 years ago

Hello

Somtimes, without knowing the reason, i have the file continue downloading after 100% ! From my download logs : [2022-04-29 02:42:42.554] [silly] 528.0 kB/s - 1596.5% [4.7 GB/304.1 MB]

How it can happens?

Thank you

hgouveia commented 2 years ago

Hello @khkassem, is this happening with different URL or only with this specific file? what could be happening is that the server is reporting wrongly the file size, ndh uses content-length header to determine the file size, if these URL are public, could you share it so i can try to reproduce myself? thanks

khkassem commented 2 years ago

Hello

I have 70 users downloading the same files, and i get this problem on 50% of them, when I download the file using chrome its ok, some times just restarting my software will continue the download correctly

The url is not public for now, i'll see if i can generate a public link with the same conditions.

Best reegards

hgouveia commented 2 years ago

Hello

I have 70 users downloading the same files, and i get this problem on 50% of them, when I download the file using chrome its ok, some times just restarting my software will continue the download correctly

The url is not public for now, i'll see if i can generate a public link with the same conditions.

Best reegards

cool, also would be very helpful to know which version are you using, what options are you using, and repro steps, so i can try to reproduced it on my end

khkassem commented 2 years ago

Hello

It seems that I have this problem when I have multiple downloads at the same time I'll add this code to the progress management :

    newDownloadHelper.on('progress', async (state: Stats) => {
      const currentTime = new Date()
      const elaspsedTime = currentTime.getTime() - startTime.getTime()
      if (elaspsedTime > 3000) {
        const speed = byteHelper(state.speed)
        const progress = state.progress.toFixed(1)
        const downloaded = byteHelper(state.downloaded)
        const total = byteHelper(state.total)
        startTime = currentTime
        logger.silly(`${speed}/s - ${progress}% [${downloaded}/${total}]`)
        //id downloaded > total break the download
        if (downloaded > total) {
          logger.error(
            `download-manager progress downloaded greater than toatl, break download ${fileServerInfo.id} ${fileServerInfo.url}`
          )
          newDownloadHelper.stop()
        }
        await Download?.updateDownload({ progress: progress, status: 2 }, fileServerInfo.id)
        if (progressCallback) progressCallback(fileServerInfo.id, state)
      }
    })

Just to stop the download when the downloaded file size is more than the file size itself.

I am using node 12 and i am not able to ugrade now. I am using node downloader helper 2.1.0, I'll try to upgrade to the latest

Thank you

hgouveia commented 1 year ago

@khkassem i released a new version v2.1.2 , could you try with this, and see if still happening?, also please make sure to create a new DownloadHelper per file

roddc commented 1 year ago

I also have this problem, it seems to happen on insufficient disk space, and if the user remove some files in the middle of download.

SplitGemini commented 1 year ago

exceed 100% +1

SplitGemini commented 1 year ago

It happened when remote file is not resumable but a resume method call is performed. Such as restarting the file server

SplitGemini commented 1 year ago

It happened when remote file is not resumable but a resume method call is performed. Such as restarting the file server

Once resume, the problem will recur, even if remote file is resumable.

SplitGemini commented 1 year ago

One situation when error occured, it retries repeatedly. image

hgouveia commented 1 year ago

fixed on #93