make query request with 20 concurrency and execute 20 times in each concurrency
After the execution is completed, wait 5 minutes before exit the process, then the log “a connection to http:// was leadked. Did you forget to close a response body” will appear.
It takes 5 minutes to wait for the log to appear because there is a connection pool cleanup task in okhttp3, and the cleanup task is executed once every 5 minutes by default. The above log is print within cleanup function, when cleanup found the connection in the pool is still in use but its reference is empty, the cleanup will mark the connection is leaked and then will print the log and clear the connection.
okhttp3 uses a connectionPool to realize connection reuse. When using the connection pool to send a request request, each connection will record the weak reference of newCall.
Record the weak reference of newCall in the connection object. (That is, record that the current connection is referenced by which newCalls)
In okhttp3, only when the response corresponding to the request is read, the newCall weak reference recorded in the connection object mentioned above will be removed.
If a response has not been read or has not been read completely, then the connection will always be occupied, but in Java, the response(or we can say responseBody) will Automatic garbage collected by jvm. After 5 minutes, the automatic cleaning mechanism of the okhttp3 connection pool will find that the connection is occupied, but the newCall of the connection's weak reference is null. So it is considered that the current connection is leaked, so the log is printed and the connection is recycled.
# resolution
Rewrite the flush and read methods in http2Client. During flush, all the data streams of the response are taken out and given to a global buffer, and then close the response. During read, batches are read from the global buffer.
# tests
* Executed concurrently 20 times, the process will automatically exit after waiting for 5 minutes, and no log will appear.
* Because the okhttpclient defined in the client is a singleton, and the connection pool of okhttp is maintained internally, in order to confirm that calling http2Client.close during the execution process does not affect the execution of the existing session, the following test was performed: (The result is that it does not affect the existing session. execution)
* Temporarily modify the code logic: when an auth failure occurs when creating session, do connection.close uniformly, regardless of sessionPool.
* The pool setting max is 20 and min is 10. 10 links will be built during initialization.
* Run a single-threaded query and sleep for 10 seconds.
* Manually delete the user during the executions.
* When running 20 concurrent queries for each 20 concurrent operations, 10 user not exit errors occurred, and the other executions were normal.
# 复现场景:
* 并发20,每个并发内执行20次query。
* 执行完成后 进程退出前等待5分钟,便会出现日志“a connection to http:// was leadked. Did you forget to close a response body”。
要等待5分钟才出现日志是因为okhttp client内有一个connection pool,pool的cleanup任务是默认5分钟执行一次。上述日志便是cleanup时发现 池子中的connection还在使用但引用者为空,便认定该connection 泄漏,于是会打印日志并将其清除。
close https://github.com/vesoft-inc/nebula-java/issues/599
reproduce:
cleanup
function, whencleanup
found the connection in the pool is still in use but its reference is empty, thecleanup
will mark the connection is leaked and then will print the log and clear the connection.If a response has not been read or has not been read completely, then the connection will always be occupied, but in Java, the response(or we can say responseBody) will Automatic garbage collected by jvm. After 5 minutes, the automatic cleaning mechanism of the okhttp3 connection pool will find that the connection is occupied, but the newCall of the connection's weak reference is null. So it is considered that the current connection is leaked, so the log is printed and the connection is recycled.
okhttp3 采用一个connectionPool实现连接的复用,在使用连接池发送request请求时 每个连接会记录newCall的弱引用, 把newCall的弱引用记录在connection对象中。(也就是记录一下当前connection 是被newCall 引用的) 在okhttp3中只有request对应的response读取完毕才会将上面所提到的connection对象所记录的newCall弱引用移除。
如果一个response没有被读取或者没有被去读完,那么这个连接就一直被占用,但是在java中response没有被占用了便会 自动GC掉,5分钟后okhttp3 连接池的自动清理机制会发现connection处于被占用状态,但是connection的newCall的弱 引用为空,则认为当前connection发生了泄漏,于是打印日志并将connection回收掉。