[Q]Force Carbonlink request?

[itinneed2022]

Is there a way to force go-carbon to store more objects in memory?

My organization has ran into this bug with the python version of carbon. Briefly, that issue is that carbon-link is failing to find datapoints in memory.

So my management wants to test go-carbon's carbonlink module to see if it can find data in memory where the original python version failed. The problem is that go-carbon works far to efficiently. Everything is immediately output to disk and Graphite can read it without issue. So I need a way to slow-down go-carbon so it has to use carbonlink. Then its just up to me to figure out how to write carbonlink's actions to a log file so they can review them.

[Civil]

You can limit amount of updates that it can do, or you can use cgroups to slowdown disk writes.

[itinneed2022]

You can limit amount of updates that it can do, or you can use cgroups to slowdown disk writes.

I tried adjusting max-updates-per-second to as little as 10 updates. Memory usage stayed at 0.2% (viewed through top, rhel8 OS).

[itinneed2022]

Cache is set as:

[cache]
# Limit of in-memory stored points (not metrics)
max-size = 100000000
# Capacity of queue between receivers and cache

[deniszh]

You can increase cache size, of course. But as far as I remember carbonlink should work nonetheless - graphite-web should check cache through carbolink then fallback to whisper read. As bit of extreme test you can set up graphite-web on the separate host.

[itinneed2022]

@deniszh It sounds like carbonlink should already be engaged regardless of whether or not data is stuck in memory? Is there any kind of logging I can enable to see when go-carbon's carbonlink method was called?

[Civil]

Memory usage stayed at 0.2% (viewed through top, rhel8 OS).

Well... percents are not very helpful without knowing how much RAM you have :)

Also, it depends on the load that your go-carbon instance has. If it is not high enough - it will quickly reach the cap and stop. And then it would be better to use blkio cgroup to throttle the process (you can limit both write_bps and write_iops).

[itinneed2022]

I have to ask a question. Is there a reason why go-carbon doesn't wait to use the full amount of memory put aside for the carbon-cache before writing to disk? My Director says it make more sense to use memory before writing to disk and the behavior we're seeing is that go-carbon writes to disk as soon as it receives a metric.

[itinneed2022]

Memory usage stayed at 0.2% (viewed through top, rhel8 OS).

Well... percents are not very helpful without knowing how much RAM you have :)

Also, it depends on the load that your go-carbon instance has. If it is not high enough - it will quickly reach the cap and stop. And then it would be better to use blkio cgroup to throttle the process (you can limit both write_bps and write_iops).

Memory on the host is 24 GB.

[deniszh]

Carbon is a data storage, so, it caches data only for necessity - when it tries to order data in the memory for better disk writing. It will flush data to the disk for consistency instead of catching it. On the other hand - if you have free memory it will be used for file cache, so, file will be read from memory instead of disk. Long story short - go-carbon will use all available memory that way or another. TBH I don't really understand what's your trying to achieve here.

[itinneed2022]

Carbon is a data storage, so, it caches data only for necessity - when it tries to order data in the memory for better disk writing. On the other hand - if you have free memory it will be used for file cache, so, file will be read from memory instead of disk. Long story short - go-carbon will use all available memory that way or another. TBH I don't really understand what's your trying to achieve here.

I think my director is concerned that if we deploy go-carbon and we haven't tested whether or not go-carbonlink will be able to pull from memory that it won't solve the problem from this bug in the original carbon.

I know its an entirely new code-base but he feels that the bug may have somehow carried over.