As a user, I don't want connection leaks - Githubissues

epag commented 3 weeks ago

Author Name: Jesse (Jesse) Original Redmine Issue: 79612, https://vlab.noaa.gov/redmine/issues/79612 Original Date: 2020-06-11

During USGS ingest after some 503 responses with retry: @2020-06-11T14:07:56.293+0000 WARN OkHttpClient A connection to https://nwis.waterservices.usgs.gov/ was leaked. Did you forget to close a response body?@

Redmine related issue(s): 99811

epag commented 3 weeks ago

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2020-06-11T14:20:23Z

There's a chance that some of my WebClient changes might fix this.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-06-11T14:43:30Z

Agree.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-23T20:47:51Z

From a job with AHPS data from WRDS:

2020-12-23T19:43:10.048+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-23T19:43:10.076+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-23T19:43:10.076+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-23T19:43:10.077+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-23T19:43:10.077+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-23T19:43:10.078+0000 WARN OkHttpClient A connection to https://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
        at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
        at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
        at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:219)
        at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
        at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
        at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
        at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
        at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
        at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-01-08T19:23:39Z

From another WRDS AHPS job:

2020-12-27T01:20:23.665+0000 WARN OkHttpClient A connection to http://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
    at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
    at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
    at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
    at wres.io.utilities.WebClient.getFromWeb(WebClient.java:240)
    at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
    at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
    at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
    at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
    at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
    at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
    at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-27T01:20:23.668+0000 WARN OkHttpClient A connection to http://nwcal-wrds.[host]/ was leaked. Did you forget to close a response body?
java.lang.Throwable: response.body().close()
    at okhttp3.internal.platform.Platform.getStackTraceForCloseable(Platform.kt:137)
    at okhttp3.internal.connection.RealCall.callStart(RealCall.kt:170)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:151)
    at wres.io.utilities.WebClient.tryRequest(WebClient.java:323)
    at wres.io.utilities.WebClient.getFromWeb(WebClient.java:240)
    at wres.io.utilities.WebClient.getFromWeb(WebClient.java:184)
    at wres.io.reading.wrds.ReadValueManager.getInputBytes(ReadValueManager.java:152)
    at wres.io.reading.wrds.ReadValueManager.save(ReadValueManager.java:177)
    at wres.io.reading.wrds.WRDSSource.save(WRDSSource.java:140)
    at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:502)
    at wres.io.concurrency.IngestSaver.execute(IngestSaver.java:30)
    at wres.io.concurrency.WRESCallable.call(WRESCallable.java:18)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-15T15:03:34Z

Hank saw this again, also with WRDS

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2024-07-05T13:26:18Z

Ongoing issue, witnessed recently too.

NOAA-OWP / wres

As a user, I don't want connection leaks #228