emgerner-msft
commented
9 years ago Thanks for the very detailed description - it's helpful! Specifying the max execution time is the correct thing to be doing.
As of version 5.1 we made a bunch of fixes to how we handle socket timeouts. So, for pre-5.1 I'd blame these bugs. That being said, you mentioned that you're still seeing issues at this point using 5.1 so I'm a little less sure what could be wrong.
I know you said it's hard to get logs given how you reproduce the issue, but would it be possible to get a stack trace that's definitely from the 5.1 version? I'll try to reproduce on my end as well but I have unfortunately good cell signal. :) This step will eliminate issues caused by the previous bug so I can track down whatever is happening now.
Could you also tell me which version of Android you see this issue on and/or if there a versions you don't see the issue on? This will help me track down if there are any changes or bugs in the Android framework which could affect this issue.
Thanks, Emily
dhakimTRX
I've noticed an issue in shutting down my app where threads running various queries through this library get stuck for upwards of 10 minutes. This appears consistent with some path through the library in which socket timeouts are not properly set.
The issue can be duplicated by running a query on a phone with no SIM card, with very poor, but still connected, WiFi. My test involves achieving this poor connectivity by either walking to the end of the hallway, or hanging out in the elevator 10 floors above my office, starting a ton of queries, then shutting down my application. This behavior cannot be duplicated by either disabling WiFi on the device or completely walking out of WiFi range.
The threads in question will not exit until A: user toggles WiFi on the device, B: user walks back into WiFi range, or C: some extremely long default timeout triggers (>10 minutes)
The locked up query threads in question are running CloudTable.execute on various simple queries against my table storage account. TableRequestOptions are all default, except for setting a maximum execution time in milliseconds of 10000. On all CloudTable instances, I have run cloudTable.getServiceClient().getDefaultRequestOptions().setRetryPolicyFactory(new RetryNoRetry()).
It is rather difficult to pull full stack traces, as the bug has generally resolved itself by the time I can return to my desk, as I am thus back within WiFi range. I have managed to pull one such stack trace however:
at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method) at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:320) at com.android.okhttp.Connection.upgradeToTls(Connection.java:1285)The above stack trace may not be exactly with the 0.5.1 release, however I can state that the issue in my app remains. I have built some tools to show the states of each thread in this frozen state. Locked up threads are generally in one of:at com.android.okhttp.Connection.connect(Connection.java:1197)
at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:392)
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:295)
at com.android.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:373) at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:323) at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:491) at com.android.okhttp.internal.http.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
at com.android.okhttp.internal.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:123)
at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
A. libcore.io.Posix.connect(Native Method), B. com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method), or C. com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method)
All threads in question have been interrupted, and their interrupted flag is set at the time of determining their stack traces.
I am currently hoping that I have missed some simple setting in the client application, (There's no handy dandy kill all open connections method is there?) or that the library has missed some socket timeout, as I truly do not want to track this to issues in Posix or NativeCrypto.
Please let me know if there is any more information you would need to track this down, and thank you for providing the library that makes my application possible,
-Dan