search

lingochamp / okdownload

A Reliable, Flexible, Fast and Powerful download engine.
Apache License 2.0
5.13k stars 767 forks source link

BUG: bulk download stuck #484

Open coccoalberto97 opened 2 years ago

coccoalberto97 commented 2 years ago

OkDownload Version

1.0.7

Problem Describe

Randomly (that happened 20% of the times so far) when trying to download files in bulk (900MB - 600 files) the queue gets stuck and it stops emitting events:

this is the code I'm using to set up the bulk download:

DownloadContext.Builder builder = new DownloadContext.QueueSet()
                        .setParentPathFile(new File(context.getCacheDir().toString() + File.separator + "tmpapp"))
                        .setMinIntervalMillisCallbackProcess(300)
                        .commit();
                for (int i = 0; i < downloadTasks.size(); i++) {
                    //downloadTasks list of MyDownloadTaskClass 
                    MyDownloadTaskClass task = downloadTasks.get(i);
                    builder.bind(task.url).addTag(0, task.filename).addTag(1, task.savedDir).addTag(2, task.finalDir).addTag(3, i);
                }

                DownloadContext downloadContext = builder.setListener(new DownloadContextListener() {

                    @Override
                    public void taskEnd(@NonNull DownloadContext context, @NonNull com.liulishuo.okdownload.DownloadTask task, @NonNull EndCause cause, @Nullable Exception realCause, int remainCount) {
                        final int index = (int) task.getTag(3);
                        synchronized (updateFileNumberLock) {
                            switch (cause) {
                                case COMPLETED:
                                    break;
                                case PRE_ALLOCATE_FAILED:
                                case FILE_BUSY:
                                case ERROR:
                                case CANCELED:
                                    log("failed at index " + index);
                                    if (index < downloadTasks.size()) {
                                        failedTasks.add(downloadTasks.get(index));
                                        failedDownloads++;
                                    }
                                    break;
                            }
                            downloadedFiles++;
                        }
                    }

                    @Override
                    public void queueEnd(@NonNull DownloadContext context) {
                        //downloadFinished variable used to exit the service loop and hide the foreground notification
                        downloadFinished = true;
                        log("QUEUE END");
                    }
                }).build();

downloadContext.start(
                        new DownloadListener() {
                            @Override
                            public void taskStart(@NonNull com.liulishuo.okdownload.DownloadTask task) {
                                Log.e("DOWNLOADCONTEXT", "taskStart "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void connectTrialStart(@NonNull com.liulishuo.okdownload.DownloadTask task, @NonNull Map<String, List<String>> requestHeaderFields) {
                                Log.e("DOWNLOADCONTEXT", "connectTrialStart "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void connectTrialEnd(@NonNull com.liulishuo.okdownload.DownloadTask task, int responseCode, @NonNull Map<String, List<String>> responseHeaderFields) {
                                Log.e("DOWNLOADCONTEXT", "connectTrialEnd "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void downloadFromBeginning(@NonNull com.liulishuo.okdownload.DownloadTask task, @NonNull BreakpointInfo info, @NonNull ResumeFailedCause cause) {
                                Log.e("DOWNLOADCONTEXT", "downloadFromBeginning "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void downloadFromBreakpoint(@NonNull com.liulishuo.okdownload.DownloadTask task, @NonNull BreakpointInfo info) {
                                Log.e("DOWNLOADCONTEXT", "downloadFromBreakpoint "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void connectStart(@NonNull com.liulishuo.okdownload.DownloadTask task, int blockIndex, @NonNull Map<String, List<String>> requestHeaderFields) {
                                Log.e("DOWNLOADCONTEXT", "connectStart "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void connectEnd(@NonNull com.liulishuo.okdownload.DownloadTask task, int blockIndex, int responseCode, @NonNull Map<String, List<String>> responseHeaderFields) {
                                Log.e("DOWNLOADCONTEXT", "connectEnd "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void fetchStart(@NonNull com.liulishuo.okdownload.DownloadTask task, int blockIndex, long contentLength) {
                                Log.e("DOWNLOADCONTEXT", "fetchStart "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void fetchProgress(@NonNull com.liulishuo.okdownload.DownloadTask task, int blockIndex, long increaseBytes) {
                                Log.d("DOWNLOADCONTEXT", "fetchProgress "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void fetchEnd(@NonNull com.liulishuo.okdownload.DownloadTask task, int blockIndex, long contentLength) {
                                Log.d("DOWNLOADCONTEXT", "fetchEnd "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }

                            @Override
                            public void taskEnd(@NonNull final com.liulishuo.okdownload.DownloadTask task, @NonNull final EndCause cause, @Nullable final Exception realCause) {
                                Log.d("DOWNLOADCONTEXT", "taskEnd "+ task.getFilename());
                                cyclesWithouUpdate = 0; //update received reset the counter
                            }
                        }, false);

                while (!downloadFinished) {
                    log("CHECK LOOP cyclesWithouUpdate " + cyclesWithouUpdate + " downloadedFiles " + downloadedFiles + " failedDownloads " + failedDownloads + " progress " + (downloadedFiles - failedDownloads) * 100 / totalFiles);
                    //after not receiving updates for 60 seconds I kill the queue and retry stuck downloads
                    if(cyclesWithouUpdate > 60){
                        log("cyclesWithouUpdate > 60 stop and retry failedTasks "+failedTasks.size());
                        downloadContext.stop();
                    }
                    setForegroundAsync(updateNotification(context, notificationName, DownloadStatus.RUNNING, (downloadedFiles - failedDownloads) * 100 / totalFiles, null));
                    cyclesWithouUpdate++;
                    Thread.sleep(1000);
                }

Log

2022-05-25 10:06:33.267 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:33.651 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:34.021 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:34.324 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:34.628 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:34.953 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.266 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.566 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.897 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.944 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchProgress f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.955 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchEnd f9b760b0-263e-4316-bc4c-8cff7b64fe9f
2022-05-25 10:06:35.955 14078-14078/com.trizeta.pandora E/DownloadAllWorker: taskEnd f9b760b0-263e-4316-bc4c-8cff7b64fe9f

//from here I stop receiving updates from tasks and "queueEnd" is never called

2022-05-25 10:06:37.694 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 1 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:38.696 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 2 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:39.698 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 3 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:40.701 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 4 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:41.703 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 5 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:42.708 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 6 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:43.710 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 7 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:44.715 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 8 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:45.718 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 9 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:46.720 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 10 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:47.727 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 11 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:48.729 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 12 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:49.734 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 13 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:50.737 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 14 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:51.745 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 15 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:52.750 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 16 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:53.755 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 17 downloadedFiles 913 failedDownloads 3 progress 99
2022-05-25 10:06:54.759 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 18 downloadedFiles 913 failedDownloads 3 progress 99

[...]

2022-05-25 10:07:37.948 14078-14675/com.trizeta.pandora D/DownloadAllWorker: CHECK LOOP cyclesWithouUpdate 61 downloadedFiles 913 failedDownloads 3 progress 99

//3 failed downloads atm
2022-05-25 10:07:37.948 14078-14675/com.trizeta.pandora D/DownloadAllWorker: cyclesWithouUpdate > 60 stop and retry 3

//stop downloads
2022-05-25 10:07:37.950 14078-14078/com.trizeta.pandora E/DownloadAllWorker: fetchEnd 283b7f6a-c96e-4e2f-916e-6edd3b49b082
2022-05-25 10:07:37.952 14078-14078/com.trizeta.pandora E/DownloadAllWorker: taskEnd 283b7f6a-c96e-4e2f-916e-6edd3b49b082
2022-05-25 10:07:37.952 14078-14078/com.trizeta.pandora D/DownloadAllWorker:failed at index 405
2022-05-25 10:07:37.952 14078-14078/com.trizeta.pandora E/DownloadAllWorker: taskEnd cf4deb64-7a7f-4f13-a54e-2306eefeeffe
2022-05-25 10:07:37.952 14078-14078/com.trizeta.pandora D/DownloadAllWorker: failed at index 442
2022-05-25 10:07:37.952 14078-14078/com.trizeta.pandora D/DownloadAllWorker: QUEUE END
//2 downloads stuck so far
KhalidElSayed commented 2 years ago

any solution here?!

midou221 commented 9 months ago

same here...