[new feature request] : The ability to resume downloading from where it left off

[shahmirzali49]

can you implement/add a new feature the ability to resume incomplete downloads and append data to an existing file?

have a look at this: https://dev.to/rlazom/resume-downloads-in-flutter-with-dio-abc

I know you are using the HTTP package in this package. can you do it with the HTTP package? (when you have time 🙂)?

[starkdmi]

@Shahmirzali-Huseynov, do you have an issue with resuming? I use my other package named download_task based on http, and it supports the resuming, as well as this package do.

[shahmirzali49]

I'm not meaning normal resume and pause functionality.

Current issue :

I was facing a problem with my current project. I need to handle large mp3 files, sometimes the download would not complete or close/exit the app and every time the user accesses this specific view the download starts again from the beginning.

the HTTP does not have the ability to append data to an existing file during the download.

[starkdmi]

@Shahmirzali-Huseynov, there is a link to my package based on http, and yes, it can append data to existing file.

Here is the code.

[shahmirzali49]

@starkdmi for example, I start downloading and its progress is 50%. at his time I close the app and after a while opened it again. Will the download start from where it left off? ( I mean from 50%) ?

[starkdmi]

@Shahmirzali-Huseynov, if the file wasn't deleted - it should.

This is one of the main features, let me know if it doesn't actually continue.

[shahmirzali49]

@starkdmi I tested your example sample. yeah, your right but for a single download.

but for multiple downloads (https://github.com/starkdmi/flutter_download_manager/issues/1#issuecomment-1592085966) it's not resumed. it starts from 0. -> https://youtube.com/shorts/q7R5RVGYwG0?feature=share

[starkdmi]

@Shahmirzali-Huseynov, I do remember you had code for cleaning the downloading directory, did you disabled it?

Let's say 20 of 100 files are downloaded, you closed the app, restart the app and downloading - you will pass 100 links to DownloadManager which will process them in queue using available Isolates, if file exists, it tries to continue downloading. Now it depends on server response, if server sent file size which is equals to downloaded - done, let's move to next link, but if your device have just a part of file - it continues.

The main thing here what you still will have summary progress 0% on restart. As you know there is no such a thing as summary progress in my code, so the logic we used to calculate it can be changed to take in calculations amount of downloaded file. For instance, I wouldn't pass the already downloaded files to the DownloadManager, I will remove those from all links when file is downloaded.

[shahmirzali49]

@starkdmi I wanted to clear when there is an error or the user didn't finish downloading. but now I wanna resume where the download left off.

[shahmirzali49]

What do you think is there a workaround or solution for multiple downloads? (https://github.com/starkdmi/flutter_download_manager/issues/1#issuecomment-1592085966)

[shahmirzali49]

by the way, I removed the clear function from init. but same starting from again(0).

 static Future<void> clean({removeDirectory = false}) async {
    // remove files in default directory
    final dir = Directory(directory);
    if (await dir.exists()) {
      await for (final file in dir.list()) {
        if (await file.exists()) {
          await file.delete();
        }
      }
      if (removeDirectory) {
        dir.delete();
      }
    }
  }

[shahmirzali49]

and what must happen if the URL/file is downloaded completed(100%) before ? this control up to us or? package doing something for it?

[starkdmi]

@Shahmirzali-Huseynov, it's totally on you, the package doesn't store the downloaded files in db, so it cannot skip already downloaded files on re-run, but will check if it's downloaded fully and continue if needed.

[starkdmi]

@Shahmirzali-Huseynov, if I re-run the downloading and the file is partially stored on the device, it continues for me. But it should be the same directory (temp directory can change between the runs).

[starkdmi]

@Shahmirzali-Huseynov, to show the correct progress in your case you can just get the amount of files stored in the directory before running the downloading. Some of them may be incomplete, but the amount of them isn't greater than the isolates amount.

[shahmirzali49]

@Shahmirzali-Huseynov, if I re-run the downloading and the file is partially stored on the device, it continues for me. But it should be the same directory (temp directory can change between the runs).

for single downloading yes, but not for multiple.

[starkdmi]

@Shahmirzali-Huseynov, the downloading is fully separated, each runs on its own Isolate in parallel. There should be no difference between one or many.

[shahmirzali49]

@starkdmi then the problem is about calculating total progress?

[shahmirzali49]

Open Ai chat gpt 4 answer to my question (file is downloaded completely 100% or not) :

You want to ensure that the MP3 file is completely downloaded before playing it. This can be a bit more challenging because the file system doesn't inherently know if the file was fully downloaded or not.

One common approach to solve this issue is by downloading the file with a temporary name or extension. Once the file is fully downloaded, you can then rename it to its final name. Here's a simple illustration of this strategy:

import 'dart:io';
import 'package:http/http.dart' as http;
import 'package:path/path.dart';

void downloadFile(String url, String filePath) async {
  var request = await HttpClient().getUrl(Uri.parse(url));
  var response = await request.close();
  var bytes = await consolidateHttpClientResponseBytes(response);
  File file = File(filePath + ".temp");
  await file.writeAsBytes(bytes);

  // Rename the file to indicate it has finished downloading
  await file.rename(filePath);
}

void playFile(String filePath) async {
  File file = File(filePath);

  // Check if the file exists, if not, it means the file hasn't finished downloading
  if (await file.exists()) {
    // Play the file
  } else {
    print("File not found, it may not have finished downloading.");
  }
}

In this code, we download the file with a .temp extension, and once the file has fully downloaded, we rename it to its final name. When you want to play the file, you first check if the file with the final name exists. If it does, you can play it. If not, it means the file hasn't finished downloading.

This is a simple solution and might not cover all edge cases, but it should give you a starting point to solve your issue.

[starkdmi]

@Shahmirzali-Huseynov, look, you can do it yourself, after downloading just move from the temporary directory to your desired one. But yeah, as I said, it's about progress calculation: 1) You should not re-run already downloaded files, which can be implemented either by storing in db which files are downloaded or not either use the temp directory and move when each file is complete. Now you use the amount of downloaded files and show the progress before downloading is restarted. 2) Restart only incomplete files, append the downloaded amount to the number from point one and show the summary progress.

[shahmirzali49]

@starkdmi you suggest to me, don't use the temporary directory?

I'm using getApplicationDocumentsDirectory

[shahmirzali49]

@starkdmi can I rename the file path like this ?

 final downloader = DownloadManager.instance;

    final request = downloader.download(my_url',
      path: 'my_path.mp3',
    );

    request.path = 'renamed.mp3';

[starkdmi]

@Shahmirzali-Huseynov, no, please rename file in events.listen() block after downloading is completed, you can use the same code from your chatGPT response.

[shahmirzali49]

@starkdmi no I meant can I rename like : request.path = 'renamed.mp3';

of course I did like what you said 🙂👍

 request.events.listen(
      (event) {
        if (event is double) {
          // Calculate the progress
          final totalProgress = event;

          state = (totalProgress);

          print("Total progress: ${totalProgress}");
        }
        if (event == DownloadState.finished) {
          print("Download finished and renamed");
          request.path = 'completed.mp3';
        }
      },
      onError: (error) {
        log("onError: $error");
      },
    );

[starkdmi]

@Shahmirzali-Huseynov, no, you can't 🫣

[shahmirzali49]

@starkdmi I mean your package doesn't provide some function ?

[starkdmi]

@Shahmirzali-Huseynov, the download method has path argument, use it to set the initial destination, which will be some temp directory in your case. Then use dart:io to rename it after downloading is complete.

[shahmirzali49]

@starkdmi Hmm Okay, I got you, thank you for your answers, effort, and time. by the way, I'm sorry to bother you 🙃😌

[starkdmi]

@Shahmirzali-Huseynov, not a problem, you're welcome 😉

[shahmirzali49]

@starkdmi there is another problem 😩 I think each download is being appended to the file. it's a problem. Because normally mp3 is 1 hour but when I resume 3,4 times it is 2,3 hour 😕.

I calculated the megabyte file size.

normal download: (file size 47 MB)

with resuming: (file size 180 MB)

my function :

Future<void> startSurahDownload({required int surahNumber}) async {
    final downloader = DownloadManager.instance;
    log("Globals.directory ${Globals.directory}");
    final String tempPath = '${Globals.directory}/abdullaah_3awwaad_al-juhaynee_$surahNumber.mp3.temp';
    final String finishedPath = '${Globals.directory}/abdullaah_3awwaad_al-juhaynee_$surahNumber.mp3';
    final request = downloader.download(
      'https://download.quranicaudio.com/quran/abdullaah_3awwaad_al-juhaynee//${getFormattedNumber(surahNumber)}.mp3',
      path: tempPath,
    );

    request.events.listen(
      (event) async {
        if (event is double) {
          // Calculate the progress
          final totalProgress = event;

          state = (totalProgress, null);

          print("Total progress: ${totalProgress}");
        }
        if (event == DownloadState.finished) {
          print("Download finished and renamed");
          File oldFilePath = File(tempPath);
          await oldFilePath.rename(finishedPath);

          ByteData data = await rootBundle.load(finishedPath);

          // Get file size in bytes
          final megabytes = data.lengthInBytes ~/ (1024 * 1024);
          state = (1.0, megabytes);
        }
      },
      onError: (error) {
        log("onError: $error");
      },
    );
  }

and interesting point in debug console logs :

1. start 0 --> end 0.4
2. start 0.28 --> end 0.35
3. start 0.32 --> end 0.53
4. start 0.44--> end 0.72
5. start 0.56 --> end 0.86
6. start 0.66 --> end 0.92
7. start 0.73 --> end 1.0

flutter: Total progress: 0.0
flutter: Total progress: 0.32
flutter: Total progress: 0.33
flutter: Total progress: 0.34
flutter: Total progress: 0.35
flutter: Total progress: 0.36
flutter: Total progress: 0.37
flutter: Total progress: 0.38
flutter: Total progress: 0.39
flutter: Total progress: 0.4
flutter: Total progress: 0.41
flutter: Total progress: 0.42
flutter: Total progress: 0.43
flutter: Total progress: 0.44
flutter: Total progress: 0.45
flutter: Total progress: 0.46
flutter: Total progress: 0.47
flutter: Total progress: 0.48
flutter: Total progress: 0.49
flutter: Total progress: 0.5
flutter: Total progress: 0.51
flutter: Total progress: 0.52
flutter: Total progress: 0.53

flutter: File not found, it may not have finished downloading.
flutter: Total progress: 0.0
flutter: Total progress: 0.44
flutter: Total progress: 0.45
flutter: Total progress: 0.46
flutter: Total progress: 0.47
flutter: Total progress: 0.48
flutter: Total progress: 0.49
flutter: Total progress: 0.5
flutter: Total progress: 0.51
flutter: Total progress: 0.52
flutter: Total progress: 0.53
flutter: Total progress: 0.54
flutter: Total progress: 0.55
flutter: Total progress: 0.56
flutter: Total progress: 0.57
flutter: Total progress: 0.58
flutter: Total progress: 0.59
flutter: Total progress: 0.6
flutter: Total progress: 0.61
flutter: Total progress: 0.62
flutter: Total progress: 0.63
flutter: Total progress: 0.64
flutter: Total progress: 0.65
flutter: Total progress: 0.66
flutter: Total progress: 0.67
flutter: Total progress: 0.68
flutter: Total progress: 0.69
flutter: Total progress: 0.7
flutter: Total progress: 0.71
flutter: Total progress: 0.72

flutter: File not found, it may not have finished downloading.
flutter: Total progress: 0.0
flutter: Total progress: 0.56
flutter: Total progress: 0.57
flutter: Total progress: 0.58
flutter: Total progress: 0.59
flutter: Total progress: 0.6
flutter: Total progress: 0.61
flutter: Total progress: 0.62
flutter: Total progress: 0.63
flutter: Total progress: 0.64
flutter: Total progress: 0.65
flutter: Total progress: 0.66
flutter: Total progress: 0.67
flutter: Total progress: 0.68
flutter: Total progress: 0.69
flutter: Total progress: 0.7
flutter: Total progress: 0.71
flutter: Total progress: 0.72
flutter: Total progress: 0.73
flutter: Total progress: 0.74
flutter: Total progress: 0.75
flutter: Total progress: 0.76
flutter: Total progress: 0.77
flutter: Total progress: 0.78
flutter: Total progress: 0.79
flutter: Total progress: 0.8
flutter: Total progress: 0.81
flutter: Total progress: 0.82
flutter: Total progress: 0.83
flutter: Total progress: 0.84
flutter: Total progress: 0.85
flutter: Total progress: 0.86

flutter: File not found, it may not have finished downloading.
flutter: Total progress: 0.0
flutter: Total progress: 0.66
flutter: Total progress: 0.67
flutter: Total progress: 0.68
flutter: Total progress: 0.69
flutter: Total progress: 0.7
flutter: Total progress: 0.71
flutter: Total progress: 0.72
flutter: Total progress: 0.73
flutter: Total progress: 0.74
flutter: Total progress: 0.75
flutter: Total progress: 0.76
flutter: Total progress: 0.77
flutter: Total progress: 0.78
flutter: Total progress: 0.79
flutter: Total progress: 0.8
flutter: Total progress: 0.81
flutter: Total progress: 0.82
flutter: Total progress: 0.83
flutter: Total progress: 0.84
flutter: Total progress: 0.85
flutter: Total progress: 0.86
flutter: Total progress: 0.87
flutter: Total progress: 0.88
flutter: Total progress: 0.89
flutter: Total progress: 0.9
flutter: Total progress: 0.91
flutter: Total progress: 0.92 

flutter: File not found, it may not have finished downloading.
flutter: Total progress: 0.0
flutter: Total progress: 0.73
flutter: Total progress: 0.74
flutter: Total progress: 0.75
flutter: Total progress: 0.76
flutter: Total progress: 0.77
flutter: Total progress: 0.78
flutter: Total progress: 0.79
flutter: Total progress: 0.8
flutter: Total progress: 0.81
flutter: Total progress: 0.82
flutter: Total progress: 0.83
flutter: Total progress: 0.84
flutter: Total progress: 0.85
flutter: Total progress: 0.86
flutter: Total progress: 0.87
flutter: Total progress: 0.88
flutter: Total progress: 0.89
flutter: Total progress: 0.9
flutter: Total progress: 0.91
flutter: Total progress: 0.92
flutter: Total progress: 0.93
flutter: Total progress: 0.94
flutter: Total progress: 0.95
flutter: Total progress: 0.96
flutter: Total progress: 0.97
flutter: Total progress: 0.98
flutter: Total progress: 0.99
flutter: Total progress: 1.0

"File not found, it may not have finished downloading." log coming from here :

final filePath = "${Globals.directory}/abdullaah_3awwaad_al-juhaynee_$surahNumber.mp3";

                          File file = File(filePath);
                          if (await file.exists()) {
                          audioPlayer.play();
                          } else { 
                           print("File not found, it may not have finished downloading.");
                           startSurahDownload(
                                  surahNumber: int.parse(surahNumber),
                                );
                          }

[starkdmi]

@Shahmirzali-Huseynov, could you try to set the safeRange to false? It's an option which reflects server response headers.

[shahmirzali49]

@starkdmi Yeah you are right, I add safeRange to false, it's working as excepted. can you explain a bit more about what is the safeRange ?

[starkdmi]

@Shahmirzali-Huseynov, whenever you want to continue downloading to existing file you need to known the full size of the file on server. This info sent to server in a Range header via bytes=$from-$size. By setting safeRange to false you allow HTTP client to use bytes=$from- header, which has no ending point. The code for this parameter is located here. The download_task package have some options to be compatible with different server configurations and this is the one of them.

[shahmirzali49]

What must happen when safeRange to true or null? @starkdmi

[starkdmi]

@Shahmirzali-Huseynov, when the safeRange is set to true, and no file length is known the Range header is not set. That's common for an initial response, not a continuing one, but may be used to enforce the file length confirmation on the server side.

[starkdmi]

@Shahmirzali-Huseynov, as for a null, the default in DownloadManager instance is true. You can set it globally for a manager or pass it to each separate download request.

[shahmirzali49]

I think My situation was safeRange true and i have fileSize content length from server. Only this time you didn't tell what should happen))? @starkdmi

[starkdmi]

@Shahmirzali-Huseynov, the Range header is calculated and sent to the server before any response. Here is the code for the safeRange, and the full code is just a 300+ lines for a resumable http client, you may scroll it to be in a context 🧑‍💻

[shahmirzali49]

sorry I was on the phone. I will look at the codes. thanks.