search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
126 stars 1.04k forks source link

Continuous GC - Allocation Failures #2703

Open eksantrik opened 5 years ago

eksantrik commented 5 years ago

My KSQL instance keeps stopping after a couple of hours and when I was troubleshooting the issue I noticed that ksql-server-gc.log.0.current file has a lot of allocation failure messages:

2019-04-17T16:02:44.673+0000: 48.286: [GC (Allocation Failure) 2019-04-17T16:02:44.674+0000: 48.286: [ParNew: 830330K->485K(929024K), 0.0097075 secs] 852477K->22634K(1961216K), 0.0098566 secs] [Times: user=0.11 sys=0.00, real=0.01 secs] 2019-04-17T16:06:05.744+0000: 249.357: [GC (Allocation Failure) 2019-04-17T16:06:05.744+0000: 249.357: [ParNew: 826341K->173K(929024K), 0.0092831 secs] 848490K->22322K(1961216K), 0.0094278 secs] [Times: user=0.10 sys=0.00, real=0.01 secs] 2019-04-17T16:11:22.731+0000: 566.344: [GC (Allocation Failure) 2019-04-17T16:11:22.731+0000: 566.344: [ParNew: 826029K->160K(929024K), 0.0092288 secs] 848178K->22310K(1961216K), 0.0093050 secs] [Times: user=0.12 sys=0.00, real=0.01 secs] 2019-04-17T16:17:02.827+0000: 906.440: [GC (Allocation Failure) 2019-04-17T16:17:02.827+0000: 906.440: [ParNew: 826016K->280K(929024K), 0.0101331 secs] 848166K->22432K(1961216K), 0.0102498 secs] [Times: user=0.09 sys=0.00, real=0.01 secs] 2019-04-17T16:23:23.056+0000: 1286.668: [GC (Allocation Failure) 2019-04-17T16:23:23.056+0000: 1286.669: [ParNew: 826136K->190K(929024K), 0.0073401 secs] 848288K->22348K(1961216K), 0.0074279 secs] [Times: user=0.07 sys=0.00, real=0.01 secs] 2019-04-17T16:26:29.922+0000: 1473.535: [GC (Allocation Failure) 2019-04-17T16:26:29.922+0000: 1473.535: [ParNew: 826046K->220K(929024K), 0.0094578 secs] 848204K->22380K(1961216K), 0.0095804 secs] [Times: user=0.11 sys=0.00, real=0.00 secs] 2019-04-17T16:32:20.526+0000: 1824.139: [GC (Allocation Failure) 2019-04-17T16:32:20.526+0000: 1824.139: [ParNew: 826076K->202K(929024K), 0.0092719 secs] 848236K->22371K(1961216K), 0.0093508 secs] [Times: user=0.11 sys=0.00, real=0.01 secs] 2019-04-17T16:38:01.489+0000: 2165.101: [GC (Allocation Failure) 2019-04-17T16:38:01.489+0000: 2165.101: [ParNew: 826058K->237K(929024K), 0.0097605 secs] 848227K->22425K(1961216K), 0.0098330 secs] [Times: user=0.11 sys=0.00, real=0.01 secs] 2019-04-17T16:43:41.627+0000: 2505.240: [GC (Allocation Failure) 2019-04-17T16:43:41.627+0000: 2505.240: [ParNew: 826093K->127K(929024K), 0.0087040 secs] 848281K->22355K(1961216K), 0.0088429 secs] [Times: user=0.10 sys=0.00, real=0.01 secs] 2019-04-17T16:50:40.791+0000: 2924.403: [GC (Allocation Failure) 2019-04-17T16:50:40.791+0000: 2924.403: [ParNew: 825983K->140K(929024K), 0.0102264 secs] 848211K->22370K(1961216K), 0.0103202 secs] [Times: user=0.11 sys=0.00, real=0.01 secs]

Does this mean that my issue is related to performance tuning? How can I resolve this?

Thanks

eksantrik commented 5 years ago

Is there anyway to optimize KSQL to avoid this error? The process keep stopping after a couple of hours constantly.

Thanks,