search

square / okhttp

Square’s meticulous HTTP client for the JVM, Android, and GraalVM.
https://square.github.io/okhttp/
Apache License 2.0
45.88k stars 9.16k forks source link

ProtocolException: unexpected end of stream while downloading a file #3715

Closed ali-idrizi closed 6 years ago

ali-idrizi commented 6 years ago

Using a library which uses okhttp to download files, I am usually getting an unexpected end of stream error while downloading large files (thrown at the end of the download). I am using the following code and URL to reproduce the error:

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder().url("http://speedtest-sfo1.digitalocean.com/100mb.test").build();
client.newCall(request).enqueue(new Callback() {
    public void onFailure(Call call, IOException e) {
        e.printStackTrace();
    }

    public void onResponse(Call call, Response response) throws IOException {
        if (!response.isSuccessful()) {
            throw new IOException("Failed to download file: " + response);
        }

        FileOutputStream fos = new FileOutputStream(downloadDirectory + File.separator + "100mb.test");
        fos.write(response.body().bytes());
        fos.close();
    }
});

URL: http://speedtest-sfo1.digitalocean.com/100mb.test (This is not the only URL the error is thrown on)

The code has been tested on API 22, 25, 26, 27. The error is not thrown always, I tested it 10 times and got the error 6 times, 4 times the file got successfully downloaded and saved..

I have read about the error and I know that it gets thrown when the servers sends incorrect Content-Length, not this time though. The file is exactly 100MB and the server supplies its correct length. I am not sending any extra request headers, the response headers are: 

Server => nginx/1.13.3
Date => Sat, 09 Dec 2017 17:19:31 GMT
Content-Type => text/plain
Content-Length => 104857600
Last-Modified => Mon, 27 Nov 2017 09:11:00 GMT
Connection => close
ETag => "5a1bd6a4-6400000"
Access-Control-Allow-Origin => *
Access-Control-Allow-Headers => Origin,X-RequestedWith,Content-Type,Range
Access-Control-Allow-Methods => GET, OPTIONS
Accept-Ranges => bytes
Cache-Control => max-age=0, no-cache, no-store, mustrevalidate, no-transform

The complete error:

Callback failure for call to http://speedtest-sfo1.digitalocean.com/...
java.net.ProtocolException: unexpected end of stream
    at okhttp3.internal.http1.Http1Codec$FixedLengthSource.read(Http1Codec.java:406)
    at okio.Buffer.writeAll(Buffer.java:1005)
    at okio.RealBufferedSource.readByteArray(RealBufferedSource.java:107)
    at okhttp3.ResponseBody.bytes(ResponseBody.java:136)
    at dm.com.downloadmanager.MainActivity$3.onResponse(MainActivity.java:273)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
    at java.lang.Thread.run(Thread.java:764)
yschimke commented 6 years ago

I haven't been able to reproduce. If you can make a reproducible test it would help a lot.

But have you considered avoiding reading into memory and streaming to the file instead? response.body().bytes() doesn't seem ideal.

ali-idrizi commented 6 years ago

Yes I am aware of that, it was just for the test (tried to make it as simple as possible), otherwise I am using a multi-thread library that does just that. The exception may not be thrown in every download, so you will have to try a few times. It is happening much more often using the library because if a thread fails the whole download does too.

I made a simple app to demonstrate the error. I changed the download URL to another one that has 10MB (same case, correct content-length), so it is faster and has to use less memory. The download stops at the last byte I think (according to the attached screenshot). I tried the app only on Android emulator API 26.

App: https://github.com/ali-idrizi/OkHttp-ProtocolException

img

swankjesse commented 6 years ago

Your test code definitely shouldn't be failing like this. Something is broken in OkHttp or the Android runtime.

yschimke commented 6 years ago

Thanks for making the sample app. I still can't get it to fail after about 30 requests

12-10 19:32:03.223 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:32:03.230 3972-3972/testaapp.okhttp.com.okhttpprotocolexception D/NetworkSecurityConfig: No Network Security Config specified, using platform default
12-10 19:32:07.785 3972-3994/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:32:08.926 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:32:13.500 3972-3999/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:32:14.448 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:32:28.192 3972-4005/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:32:29.690 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:32:34.972 3972-4011/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:32:36.237 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:32:40.833 3972-4017/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:34:03.427 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:34:03.630 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:34:04.001 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:34:04.394 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:34:04.602 3972-3972/testaapp.okhttp.com.okhttpprotocolexception I/Main: start
12-10 19:34:12.827 3972-4062/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:34:13.526 3972-4060/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:34:13.617 3972-4066/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:34:14.030 3972-4069/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
12-10 19:34:14.391 3972-4068/testaapp.okhttp.com.okhttpprotocolexception I/Main: bytes 10485760
ali-idrizi commented 6 years ago

I have no idea why this is happening. I just run it again and got the exception on the first request. I tried different OkHttp versions and compile options, the same thing happens over and over again.

12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp: Callback failure for call to http://speedtest-sfo1.digitalocean.com/...
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp: java.net.ProtocolException: unexpected end of stream
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okhttp3.internal.http1.Http1Codec$FixedLengthSource.read(Http1Codec.java:406)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okio.Buffer.writeAll(Buffer.java:1005)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okio.RealBufferedSource.readByteArray(RealBufferedSource.java:107)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okhttp3.ResponseBody.bytes(ResponseBody.java:136)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at testapp.okhttp.com.okhttpprotocolexception.MainActivity$1.onResponse(MainActivity.java:56)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
12-10 21:00:13.638 482-649/testaapp.okhttp.com.okhttpprotocolexception D/OkHttp:     at java.lang.Thread.run(Thread.java:764)

Its worth noting that there is also different people having these issues like #3682, and some also at the library I'm using https://github.com/lingochamp/FileDownloader/issues

swankjesse commented 6 years ago

@ali-idrizi does this occur on all devices?

ali-idrizi commented 6 years ago

I just tried it on a macOS and it does not happen, I also tried it on a real device running Android 5.1 and it doesn't either. So I am not sure what is the problem here, maybe the emulators?

Aaron8899 commented 6 years ago

I had similar issue when file upload, but I just change logging level with basic from body then it works fine.

swankjesse commented 6 years ago

Which devices does this occur on? X86 or ARM emulator? Running on which OS? We may end up finding a bug in a layer under OkHttp.

Aaron8899 commented 6 years ago

In my case, It occurs on Android OS in Galaxy S7 and I don't test other devices. I'm using OkHttp through Fast Android Network.

ali-idrizi commented 6 years ago

For me it is happening on all devices I have tested it on this laptop, all of them are x86. I am running Windows 10 Pro. I will test the app on an ARM emulator and update.

ali-idrizi commented 6 years ago

It does happen on ARM too, I tried it on different network/router, same story. So I guess there is something wrong with my system.

yschimke commented 6 years ago

@ali-idrizi I was really interested that you could reproduce so easily with your sample app that doesn't do anything fancy. The threading of the FileDownloader library means there are more places the bug could be, but your sample app was a normal download.

ali-idrizi commented 6 years ago

@yschimke I know, but exactly the same app is throwing an exception when I try it. I am not doing anything else except debugging the app on an API 26 x86 emulator and the exception is thrown just like in the screenshot above.

I firstly thought it was specific to some URLs, but later on I realized it happens on every URL and emulator from my system.

yschimke commented 6 years ago

Does it happen without breakpoints?

ali-idrizi commented 6 years ago

Yes it does.

Javisilox commented 6 years ago

Hi, I'm getting the same exception uploading a video of 100 kb, the content lenght seems right, I'm working with a Samsung Galaxy S6 with Android 6.0.1.

Did you find any soution?

Edit to say that in my case it was a stupid mistake and not a bug, my video file was corrupted...

Shelmanxie commented 6 years ago

HI ,I got the same problem on STB ,Android version is 4.4.2. the content length is right, While downloading a file with length of 319318529 ,it stops with this error at 40% downloading. Did you find the solution to this problem?

ali-idrizi commented 6 years ago

No, in my case it was a problem with my laptop, and it still happens, but it works pretty good on other devices I have tried.

Shelmanxie commented 6 years ago

@swankjesse @ali-idrizi ： In my case , it works fine with server on Apache Server 2.2 . But when it comes to Apache Server 2.4, the problem occurs. Hope this could help you find the solution to the problem.

swankjesse commented 6 years ago

Unsure what action to take on this. An executable test case would be really handy.

Shelmanxie commented 6 years ago

Hi , I write a demo on Android platform,just a simple download action . You can click the button to trigger it.But the button click action dose nothing currently . You may pass your own remote file URL and other parameters. Hope this would help you ,3ks in advance. https://github.com/Shelmanxie/OkhttpBugUnExceptedDemo

yschimke commented 6 years ago

@Shelmanxie If it's useful I've got a test app https://github.com/yschimke/okhttp-testapp you can build on also.

image

yschimke commented 6 years ago

@Shelmanxie I changed the app to have sensible default url and download paths, but it doesn't reproduce the problem on Android 16 or 25 emulators. I'm closing this for now, please reopen with a reproduction for us.

BTW I do appreciate the test app, thanks for doing that.

Shelmanxie commented 6 years ago

@yschimke Hi, you may use a file that its size is beyond 300M and the server should be Apache Server 2.4. Try it, and you may reproduce it. Android version may not be the cause of this problem.

yschimke commented 6 years ago

OK, trying your app with https://speed.hetzner.de/1GB.bin

If it passes for me, we will be back to needing some logs, including things like proxies that are in use. Maybe it's something like data caps. n.b. Does it fail for you on https addresses as well, I've been mostly testing with secure traffic.

Shelmanxie commented 6 years ago

As for https address, we didn't use it in our business scene. As I mentioned above, file size which is beyond 300M and Apache Server 2.4 cause this problem.

yschimke commented 6 years ago

OK, I don't have an Apache Server 2.4 with a 300M file available to me. Please reopen if you can provide one and I'll test with it.

Shelmanxie commented 6 years ago

Sorry ,I am afraid that I can't provide one for you, cause I don't own a public IP address or a remote server. So I test it with my own computer providing a local Apache 2.4 server .I do have a Apache server 2.4 software package available,but the upload speed to github in my country is too too slow because of the network wall,there's no possibility to have a success upload of the package. Maybe you need to make a local server by yourself . Sorry again...

japer21 commented 6 years ago

I have the same issue over and over again. I am using Android emultor, Pixel API27, Android 8.1, CPU x86 and the version of okhttp is 3.10.0

aimran50 commented 6 years ago

I too am having the exact same issue. I am using HttpURLConnection as documented here: https://developer.android.com/reference/java/net/HttpURLConnection.

The app makes lots of connections and transfers small amount of data(in kb range).

After about 4 to 5 requests, the "unexpected end of stream error" is seen. Everything halts for some time and then resumes.

Happens on hardware as well as emulator. Emulator: Android 8.1 , API 27 Phone: Android 8.0 Do not see this error on API 23. So, it appears something added after that version is causing this.

Here's the stacktrace: Caused by: java.net.ProtocolException: unexpected end of stream 06-27 14:19:04.306 3854-3956/com.myapp.android W/System.err: at com.android.okhttp.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:396) at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:371) at java.io.InputStream.read(InputStream.java:101)

Perhaps this should be reopened and looked at again?

swankjesse commented 6 years ago

@aimran50 can you provide an executable test case?

aimran50 commented 6 years ago

Unfortunately I dont have a executable that I can send. Its all buried inside the main app. So many people have reported this with exact same problem. Generally, when there is smoke there is a fire. You may want to version compare what changed between the Android release where it worked and the reported release where it didn't.

Imran

From: Jesse Wilson notifications@github.com Sent: Saturday, June 30, 2018 8:46:39 AM To: square/okhttp Cc: aimran50; Mention Subject: Re: [square/okhttp] ProtocolException: unexpected end of stream while downloading a file (#3715)

@aimran50https://github.com/aimran50 can you provide an executable test case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/square/okhttp/issues/3715#issuecomment-401542280, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYx42kmlrEa9yDg2Jg4vZwtZQN5r5pdjks5uB4E_gaJpZM4Q8LV2.

andrewleech commented 6 years ago

I'm very regularly getting the same error in an app using the HERE SDK (mapping library) https://www.developer.here.com/documentation/android-premium/dev_guide/topics/maps.html

In case it helps, here is the stack trace:

E/NetworkProtocol(  375): NetworkProtocol::GetTask::run exception: java.net.ProtocolException: unexpected end of stream
W/System.err(  375): java.net.ProtocolException: unexpected end of stream
W/System.err(  375):    at com.android.okhttp.internal.http.HttpConnection$FixedLengthSource.read(HttpConnection.java:449)
W/System.err(  375):    at com.android.okio.RealBufferedSource$1.read(RealBufferedSource.java:168)
W/System.err(  375):    at java.io.BufferedInputStream.read(BufferedInputStream.java:290)
W/System.err(  375):    at java.io.InputStream.read(InputStream.java:162)
W/System.err(  375):    at com.here.network.NetworkProtocol$GetTask.doInBackground(NetworkProtocol.java:418)
W/System.err(  375):    at com.here.network.NetworkProtocol$GetTask.doInBackground(NetworkProtocol.java:181)
W/System.err(  375):    at android.os.AsyncTask$2.call(AsyncTask.java:327)
W/System.err(  375):    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
W/System.err(  375):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
W/System.err(  375):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
W/System.err(  375):    at java.lang.Thread.run(Thread.java:818)

This is on an android device whose only internet connection is via a bluetooth tether to android phone, so is rather slow. I haven't seen the issue in an app using the same version of the HERE SDK running on the android phone (with fast wifi connection)

swankjesse commented 6 years ago

@andrewleech sounds like a bug in the connection between client and server. The client isn't reading fast enough so the server truncates the response.

LightSun commented 6 years ago

@swankjesse i agree . i have the problem too. the file size is 51KB.

emileb commented 5 years ago

Has anyone found solution to this? I can cause it if I artifically slow down the download. On API 28 downloading a 30MB file from my server. Need to get the fixed ASAP as causing issues for user. Any help appreciated!

Shelmanxie commented 5 years ago

hello ,I fix this by changing Apache configuration. just like the picture shows below. image

Shelmanxie commented 5 years ago

so ,it seems like a server problem.

thesiamak commented 4 years ago

I had the exact same problem with uploading small files to server and it seems solved by changing logging level! Strange! AndroidNetworking.enableLogging(HttpLoggingInterceptor.Level.BODY); To AndroidNetworking.enableLogging(HttpLoggingInterceptor.Level.BASIC);

VijayBest commented 4 years ago

I facing the same problem and use MAC OS and test it on Android 10

rainyandsunny commented 4 years ago

I facing the same problem on Honor 8X with Android 8.1.0

LiarrDev commented 4 years ago

Same issue.

I'm using: OkHttp 4.8.0 Honor 10 Lite with Android 10

DileepKumarkillari471 commented 4 years ago

any one know the solution, please tell ME

rakesh-krishna commented 3 years ago

I am also facing the same problem not sure how to handle it. I am running a Flask Api

skyzhw commented 3 years ago

In a specific series of models + a specific server + a specific Wifi network, the probability of this problem is very high, and it is also related to the network usage at the time. The problem I encountered is that the probability of occurrence is very high in Huawei mobile phones, Alibaba Cloud CDN and my company's Wifi. After verification, it is found that the default value of the ReceiveBufferSize of the socket in the Huawei system is 2097152, and the default value of the Samsung system is 1048576. The guess is related to the system tuning of this parameter. Adjust this parameter to below 49152, the probability of this problem is very low, and it is almost difficult to reproduce. You can also try if you encounter similar problems. Core code: new OkHttpClient.Builder().socketFactory(), socket.setReceiveBufferSize(49152);

OnClickListener2048 commented 2 years ago

parameter

no use

colorthoro commented 2 years ago

It seems that capacitor uses okhttp too, the ProtocolException would be thrown( almost every time) strangly only on RUN mode, everything works perfectly on DEBUG mode( in Android Studio; the emulator is Android 11, x86). V/Capacitor/Plugin: To native (Cordova plugin): callbackId: CordovaHttpPlugin1978864772, service: CordovaHttpPlugin, action: get, actionArgs: ["http:\/\/10.0.2.2:5000\/scanMusic",{"Connection":"close"},60,60,true,"json",3] W/Cordova-Plugin-HTTP: Generic request error com.silkimen.http.HttpRequest$HttpRequestException: java.net.ProtocolException: unexpected end of stream at com.silkimen.http.HttpRequest$Operation.call(HttpRequest.java:641) at com.silkimen.http.HttpRequest.copy(HttpRequest.java:2499) at com.silkimen.http.HttpRequest.receive(HttpRequest.java:1856) at com.silkimen.cordovahttp.CordovaHttpBase.processResponse(CordovaHttpBase.java:205) at com.silkimen.cordovahttp.CordovaHttpBase.run(CordovaHttpBase.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:923) Caused by: java.net.ProtocolException: unexpected end of stream at com.android.okhttp.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:398) at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:372) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:347) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.silkimen.http.HttpRequest$6.run(HttpRequest.java:2492) at com.silkimen.http.HttpRequest$6.run(HttpRequest.java:2486) at com.silkimen.http.HttpRequest$Operation.call(HttpRequest.java:635) at com.silkimen.http.HttpRequest.copy(HttpRequest.java:2499)  at com.silkimen.http.HttpRequest.receive(HttpRequest.java:1856)  at com.silkimen.cordovahttp.CordovaHttpBase.processResponse(CordovaHttpBase.java:205)  at com.silkimen.cordovahttp.CordovaHttpBase.run(CordovaHttpBase.java:84)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)  at java.lang.Thread.run(Thread.java:923)  E/Capacitor/Console: File: http://localhost/js/app.82b8407c.js - Line 1 - Msg: There was an error with the request: unexpected end of stream