rpeg
commented
4 years ago Update: The same occurred with Spotify. Not sure if it was actually due to throttling, but in any case, we aren't handling connection errors fully. Stack trace:
E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-3 Process: fho.kdvs, PID: 24002 javax.net.ssl.SSLException: Read error: ssl=0x79e74ac888: I/O error during system call, Software caused connection abort at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method) at com.android.org.conscrypt.NativeSsl.read(NativeSsl.java:399) at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:546) at okio.Okio$2.read(Okio.java:140) at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:354) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:200) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:764) Suppressed: java.net.ConnectException: Failed to connect to api.spotify.com/35.186.224.53:443 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:248) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ... 16 more Caused by: java.net.ConnectException: failed to connect to api.spotify.com/35.186.224.53 (port 443) from /192.168.50.9 (port 49584) after 10000ms: isConnected failed: ECONNABORTED (Software caused connection abort) at libcore.io.IoBridge.isConnected(IoBridge.java:278) at libcore.io.IoBridge.connectErrno(IoBridge.java:187) at libcore.io.IoBridge.connect(IoBridge.java:129) at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:137) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:390) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:436) at java.net.Socket.connect(Socket.java:621) at okhttp3.internal.platform.AndroidPlatform.connectSocket(AndroidPlatform.java:73) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ... 21 more E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-11 Process: fho.kdvs, PID: 24002 javax.net.ssl.SSLException: Read error: ssl=0x79e45cc188: I/O error during system call, Software caused connection abort at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method) at com.android.org.conscrypt.NativeSsl.read(NativeSsl.java:399) at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:546) at okio.Okio$2.read(Okio.java:140) at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:354) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:200) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:764) E/AndroidRuntime: Caused by: android.system.ErrnoException: isConnected failed: ECONNABORTED (Software caused connection abort) at libcore.io.IoBridge.isConnected(IoBridge.java:267) ... 31 more Suppressed: java.net.ConnectException: Failed to connect to accounts.spotify.com/104.154.127.47:443 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:248) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ... 16 more Caused by: java.net.ConnectException: failed to connect to accounts.spotify.com/104.154.127.47 (port 443) from /:: (port 0) after 10000ms: connect failed: ENETUNREACH (Network is unreachable) at libcore.io.IoBridge.connect(IoBridge.java:137) at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:137) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:390) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:436) at java.net.Socket.connect(Socket.java:621) at okhttp3.internal.platform.AndroidPlatform.connectSocket(AndroidPlatform.java:73) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246) ... 21 more Caused by: android.system.ErrnoException: connect failed: ENETUNREACH (Network is unreachable) at libcore.io.Linux.connect(Native Method) at libcore.io.BlockGuardOs.connect(BlockGuardOs.java:118) at libcore.io.IoBridge.connectErrno(IoBridge.java:168) at libcore.io.IoBridge.connect(IoBridge.java:129) ... 29 more
If the album art request limit is reached (which may occur with some frequency if many users are concurrently polling coverartarchive.org), a fatal SocketTimeoutException may happen. This may apply to other third party APIs as well, but I've only seen this happen with coverartarchive. We'll probably want a safeguard for connection issues.
Additionally, our observers are firing much more often than they should, which is the likely culprit for rate limit excesses. The observers fire on each database field update, and so we're making redundant polls to the APIs. (E.g. the topAdds and topAlbums on Home are being repolled when we update their DB entries with fetched info). This can probably be fixed with boolean guards around our observers that are set to true upon the first non-empty observation.
Stack-trace isn't super informative but the issue seems straightforward enough:
"E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-9 Process: fho.kdvs, PID: 8520 java.net.SocketTimeoutException: failed to connect to ia803006.us.archive.org/207.241.233.26 (port 80) from /192.168.0.208 (port 44910) after 9999ms"
https://wiki.musicbrainz.org/XML_Web_Service/Rate_Limiting