Very high memory consumption

claudio-benfatto commented 7 years ago

Hello here,

we are using the carbonzipper version 0.7.2, in combination with the carbonapi (0.8.0) and go-carbon (0.9.1).

Our cluster is constituted by 3 nodes proxied by the carbon-c-relay.

We recently upgraded from an old version of the stack, and since then we are seeing a very high memory consumption pattern:

pmap shows total 44201100K, and top around 25% usage on a very beefy server.

This is our current configuration:

carbonzipper

maxProcs: 16

timeouts:
    global: "10s"
    afterStarted: "2s"

concurrencyLimit: 0
maxIdleConnsPerHost: 100
expireDelaySec: 10

carbonapi

concurency: 20
cache:
   type: "mem"
   size_mb: 1024
   defaultTimeoutSec: 60

cpus: 2
tz: ""

sendGlobsAsIs: true

maxBatchSize: 1000

any suggestion on how to best approach the problem?

Many thanks!

dgryski commented 7 years ago

Reducing memory consumption is one thing we're working on at the moment. Reducing the maxBatchSize will help, as that will turn some very large requests that fetch a huge amount of data into a set of smaller requests that can be more easily garbage collected with similar huge spike.

claudio-benfatto commented 7 years ago

Thanks for the prompt answer @dgryski , we'll tune the settings and see if it helps.

Just a question, any reason in particular why the previous versions were not showing this issue? Or at least not up to this magnitude...

Has the caching strategy changed for the carbonzipper process?

Thanks again!

dgryski commented 7 years ago

The sendGlobsAsIs and maxBatchSize are new and still subject to tuning. It helps query latency in the majority of cases, but the down size is that it can occasionally massively increase memory usage :/

go-graphite / carbonzipper

Very high memory consumption #47