lingochamp / FileDownloader

Multitask、MultiThread(MultiConnection)、Breakpoint-resume、High-concurrency、Simple to use、Single/NotSingle-process
Apache License 2.0
11.04k stars 2.2k forks source link

360安全云盘的文件无法下载:FileDownloadGiveUpRetryException #1131

Open congwiny opened 6 years ago

congwiny commented 6 years ago

Before Issue

  1. Please search on the Issues
  2. Please search on the wiki
  3. Please set FileDownloadLog.NEED_LOG=true and review the Logcat output from main process and :filedownloader process ( pay attention to Warn and Error level logcat)

Issue

  1. What problem do you get? 下载360安全云盘的文件: 下载资源目录 下载【薛之谦-我终于成了别人的女人】下载失败
  2. Which version of FileDownloader are you using when you produce such problem? 最新master分支的代码 v1.7.5
  3. How to reproduce such problem? 使用多线程下载360安全云盘【薛之谦-我终于成了别人的女人】下载失败 下载链接可以使用QQ浏览器或UC浏览器复制已下载的此文件的URL,有可能过段时间会失效,重新下载复制URL即可。
  4. Do you set FileDownloadLog.NEED_LOG=true? YES
  5. Could you please reproduce this problem and provide all main process and :filedownloader process logcat
    main process
    
    10-12 19:22:11.663 6250-6258/com.liulishuo.filedownloader.demo I/zygote: Do partial code cache collection, code=10KB, data=22KB
    10-12 19:22:11.670 6250-6258/com.liulishuo.filedownloader.demo I/zygote: After code cache collection, code=10KB, data=22KB
    Increasing code cache capacity to 128KB
    10-12 19:22:11.674 6250-6258/com.liulishuo.filedownloader.demo I/zygote: Do partial code cache collection, code=10KB, data=40KB
    10-12 19:22:11.675 6250-6258/com.liulishuo.filedownloader.demo I/zygote: After code cache collection, code=10KB, data=40KB
    Increasing code cache capacity to 256KB
    10-12 19:22:11.678 6250-6258/com.liulishuo.filedownloader.demo I/zygote: Compiler allocated 4MB to compile void android.widget.TextView.<init>(android.content.Context, android.util.AttributeSet, int, int)
    10-12 19:22:12.102 6250-6250/com.liulishuo.filedownloader.demo I/Choreographer: Skipped 62 frames!  The application may be doing too much work on its main thread.
    10-12 19:22:12.582 6250-6258/com.liulishuo.filedownloader.demo I/zygote: Do full code cache collection, code=125KB, data=90KB
    After code cache collection, code=125KB, data=65KB
    10-12 19:22:13.569 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 8. Thread differ : +1. unknown: +1 [(+)queued-work-looper]
    10-12 19:22:13.575 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 8. Thread differ : +1. unknown: +1 [(+)queued-work-looper]
    10-12 19:22:13.577 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 8. unknown: 8
    10-12 19:22:13.944 6250-6258/com.liulishuo.filedownloader.demo I/zygote: Do partial code cache collection, code=126KB, data=72KB
    10-12 19:22:13.945 6250-6258/com.liulishuo.filedownloader.demo I/zygote: After code cache collection, code=126KB, data=72KB
    Increasing code cache capacity to 512KB
    10-12 19:22:15.690 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 12. Thread differ : +4. FlowSingle: +1 [(+)FileDownloader-Flow-01] | Event: +1 [(+)FileDownloader-EventPool1] | LauncherTask: +2 [(+)FileDownloader-LauncherTask1, (+)FileDownloader-LauncherTask2]
    10-12 19:22:15.695 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 12. Thread differ : +4. FlowSingle: +1 | Event: +1 | LauncherTask: +2
    10-12 19:22:15.698 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 12. FlowSingle: 1 | Event: 1 | LauncherTask: 2 | unknown: 8
    10-12 19:22:24.469 6250-6264/com.liulishuo.filedownloader.demo I/zygote: Background concurrent copying GC freed 242(299KB) AllocSpace objects, 0(0B) LOS objects, 49% free, 2MB/4MB, paused 783us total 215.971ms
    10-12 19:22:24.490 6250-6250/com.liulishuo.filedownloader.demo W/System.err: com.liulishuo.filedownloader.exception.FileDownloadGiveUpRetryException: require range[9534279-19068557) with contentLength(9534279), but the backend response contentLength is 5242880 on downloadId[1947487790]-connectionIndex[1], please ask your backend dev to fix such problem.
    10-12 19:22:24.492 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at com.liulishuo.filedownloader.download.FetchDataTask.run(FetchDataTask.java:105)
    10-12 19:22:24.494 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at com.liulishuo.filedownloader.download.DownloadRunnable.run(DownloadRunnable.java:121)
    10-12 19:22:24.495 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:457)
    10-12 19:22:24.497 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    10-12 19:22:24.498 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
    10-12 19:22:24.500 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
    10-12 19:22:24.501 6250-6250/com.liulishuo.filedownloader.demo W/System.err:     at java.lang.Thread.run(Thread.java:764)
    10-12 19:22:24.530 6250-6250/com.liulishuo.filedownloader.demo W/StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
    10-12 19:22:24.548 6250-6250/com.liulishuo.filedownloader.demo I/chatty: uid=10162(com.liulishuo.filedownloader.demo) identical 3 lines
    10-12 19:22:24.550 6250-6250/com.liulishuo.filedownloader.demo W/StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
    10-12 19:22:30.160 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 9. Thread differ : -3. Event: -1 [(-)FileDownloader-EventPool1] | LauncherTask: -2 [(-)FileDownloader-LauncherTask1, (-)FileDownloader-LauncherTask2]
    10-12 19:22:30.165 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 9. Thread differ : -3. Event: -1 | LauncherTask: -2
    10-12 19:22:30.167 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 9. FlowSingle: 1 | unknown: 8
    10-12 19:22:40.450 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 8. Thread differ : -1. FlowSingle: -1 [(-)FileDownloader-Flow-01]
    10-12 19:22:40.459 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 8. Thread differ : -1. FlowSingle: -1
    10-12 19:22:40.465 6250-6287/com.liulishuo.filedownloader.demo D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 8. unknown: 8
    10-12 19:29:52.724 6250-6264/com.liulishuo.filedownloader.demo I/zygote: Background concurrent copying GC freed 64323(2MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 2MB/4MB, paused 142us total 905.575ms
    10-12 19:37:30.355 6250-6264/com.liulishuo.filedownloader.demo I/zygote: Background concurrent copying GC freed 10781(600KB) AllocSpace objects, 0(0B) LOS objects, 49% free, 3MB/7MB, paused 250us total 369.721ms
    10-12 19:50:39.280 6250-6264/com.liulishuo.filedownloader.demo I/zygote: Background concurrent copying GC freed 0(50KB) AllocSpace objects, 0(0B) LOS objects, 49% free, 6MB/13MB, paused 3.122ms total 757.206ms
##### filedownloader process

10-12 19:22:14.571 6651-6651/? I/zygote: Late-enabling -Xcheck:jni 10-12 19:22:15.019 6651-6679/com.liulishuo.filedownloader.demo:filedownloader D/NetworkSecurityConfig: No Network Security Config specified, using platform default 10-12 19:22:16.794 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 16. Thread differ : +16. OkHttp: +1 [(+)OkHttp ConnectionPool] | okio: +1 [(+)Okio Watchdog] | Network: +1 [(+)FileDownloader-Network1] | FlowSingle: +1 [(+)FileDownloader-Flow-01] | Connection: +3 [(+)FileDownloader-ConnectionBlock3, (+)FileDownloader-ConnectionBlock2, (+)FileDownloader-ConnectionBlock1] | RemitHandoverToDB: +1 [(+)FileDownloader-RemitHandoverToDB] | unknown: +8 [(+)source-status-callback, (+)FinalizerDaemon, (+)HeapTaskDaemon, (+)ThreadDebugger, (+)main, (+)queued-work-looper, (+)ReferenceQueueDaemon, (+)FinalizerWatchdogDaemon] 10-12 19:22:16.796 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 16. Thread differ : +16. OkHttp: +1 | okio: +1 | Network: +1 | FlowSingle: +1 | Connection: +3 | RemitHandoverToDB: +1 | unknown: +8 [(+)source-status-callback, (+)FinalizerDaemon, (+)HeapTaskDaemon, (+)ThreadDebugger, (+)main, (+)queued-work-looper, (+)ReferenceQueueDaemon, (+)FinalizerWatchdogDaemon] 10-12 19:22:16.798 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 16. OkHttp: 1 | okio: 1 | Network: 1 | FlowSingle: 1 | Connection: 3 | RemitHandoverToDB: 1 | unknown: 8 10-12 19:22:24.899 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 15. Thread differ : -1. unknown: -1 [(-)source-status-callback] 10-12 19:22:24.902 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 15. Thread differ : -1. unknown: -1 [(-)source-status-callback] drawUpEachThreadSize: Thread count = 15. OkHttp: 1 | okio: 1 | Network: 1 | FlowSingle: 1 | Connection: 3 | RemitHandoverToDB: 1 | unknown: 7 10-12 19:22:37.031 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 14. Thread differ : -1. Connection: -1 [(-)FileDownloader-ConnectionBlock2] 10-12 19:22:37.033 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 14. Thread differ : -1. Connection: -1 10-12 19:22:37.034 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 14. OkHttp: 1 | okio: 1 | Network: 1 | FlowSingle: 1 | Connection: 2 | RemitHandoverToDB: 1 | unknown: 7 10-12 19:22:39.046 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 10. Thread differ : -4. Network: -1 [(-)FileDownloader-Network1] | FlowSingle: -1 [(-)FileDownloader-Flow-01] | Connection: -2 [(-)FileDownloader-ConnectionBlock3, (-)FileDownloader-ConnectionBlock1] 10-12 19:22:39.051 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 10. Thread differ : -4. Network: -1 | FlowSingle: -1 | Connection: -2 10-12 19:22:39.052 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSize: Thread count = 10. OkHttp: 1 | okio: 1 | RemitHandoverToDB: 1 | unknown: 7 10-12 19:28:16.469 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadInfoDiff: Thread count = 9. Thread differ : -1. OkHttp: -1 [(-)OkHttp ConnectionPool] 10-12 19:28:16.470 6651-6670/com.liulishuo.filedownloader.demo:filedownloader D/FileDownloadApplication: drawUpEachThreadSizeDiff: Thread count = 9. Thread differ : -1. OkHttp: -1 drawUpEachThreadSize: Thread count = 9. okio: 1 | RemitHandoverToDB: 1 | unknown: 7 10-12 19:31:50.957 6651-6656/com.liulishuo.filedownloader.demo:filedownloader I/zygote: Do partial code cache collection, code=30KB, data=21KB 10-12 19:31:50.958 6651-6656/com.liulishuo.filedownloader.demo:filedownloader I/zygote: After code cache collection, code=29KB, data=21KB Increasing code cache capacity to 128KB


6. Can you fix it by yourself and request PR, if not, what's problem do you get when you try to fix it

* debug跟踪了一下代码
![1](https://user-images.githubusercontent.com/9621271/46868713-c1c12e00-ce18-11e8-9293-2d4d1654714d.jpg)
![2 1](https://user-images.githubusercontent.com/9621271/46868720-c4bc1e80-ce18-11e8-9a2c-104340c563de.jpg)
![2](https://user-images.githubusercontent.com/9621271/46868724-c71e7880-ce18-11e8-9c36-762b3f8f8998.jpg)
![3](https://user-images.githubusercontent.com/9621271/46868727-cbe32c80-ce18-11e8-9e94-ec9e2613de6f.jpg)
![4](https://user-images.githubusercontent.com/9621271/46868732-cede1d00-ce18-11e8-9a4b-33abbc27ae43.jpg)
![5](https://user-images.githubusercontent.com/9621271/46868733-d1407700-ce18-11e8-9f28-3110058df0eb.jpg)

有以下发现,在分块下载部分,给服务器传的Range无效,服务器还是返回自己的Range。
这一点我在Postman上测试了一下,确实如此。
![6](https://user-images.githubusercontent.com/9621271/46868930-ad316580-ce19-11e8-9894-a96374c33316.jpg)

然后,我使用UC浏览器和QQ浏览器都能正常下载。我用Fiddler抓包试了下,看了下他们的分块。

* UC浏览器分块:
![8](https://user-images.githubusercontent.com/9621271/46869338-ed451800-ce1a-11e8-91bb-6925197072fd.jpg)

* QQ浏览器分块:
![9](https://user-images.githubusercontent.com/9621271/46869340-f0d89f00-ce1a-11e8-8ae8-4d6482bfbc6f.jpg)

发现他们刚开始都发了一个Range: bytes=0- 的请求。而且都有5242880这个块。
![10](https://user-images.githubusercontent.com/9621271/46870124-4dd55480-ce1d-11e8-87d8-807f182f5cbe.jpg)

我猜测他们自己也提前分好块了,然后请求服务器得到的分块和自己的分块数据不一致,优先又服务器的分块,然后对自己剩下的块再分配,然后再请求服务器验证分块是不是一致,如果不一致,就用服务器的分块。不知道我的猜测正不正确。

所以请教一下怎么能解决这个下载的问题~ 谢谢!
congwiny commented 6 years ago

@Jacksgong @rantianhua 我对这个问题提了个PR #1134,麻烦review一下是否有问题? 谢谢

rantianhua commented 6 years ago

抱歉回复的晚了,pull request 已经回复了。