search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.29k stars 226 forks source link

Large File System Page Cache Spike #998

Closed aronbierbaum closed 3 months ago

aronbierbaum commented 3 months ago

Occasionally the ClickHouse Backup container has massive memory increases and the memory doesn't appear to ever be released. Is there a configuration setting that I'm not setting correctly? Most times the memory increases to about 6 GB and quickly drops to the baseline of around 98 MB. Occasionally it spikes to 50GB+ and doesn't ever release the memory.

aronbierbaum commented 3 months ago

Running cat /proc/meminfo shows the following:

Cached: 59536936 kB
Slach commented 3 months ago

cached is not a processes memory this is file system page cache

watch to RSS

aronbierbaum commented 2 months ago

@Slach thank you for your response. I realize that it is a file system page cache. Is there a way to prevent it from growing so large during a backup, and/or releasing some of it when the backup is complete? Also what do you mean "watch to RSS"

Slach commented 2 months ago

RSS - resident process memory ps -ww -e -o pid,rss,comm | grep backup

it also could have spike, but it returns to system after upload or download complete you can control it with

general:
  upload_concurrency: 1

but in this case, upload will slow

Slach commented 2 months ago

Is there a way to prevent it from growing so large during a backup, and/or releasing some of it when the backup is complete

Page Cache managed by kernel, not by process.

It increased, because you need read whole database from disk during upload This is shared case for all processes, clickhouse-server also will use the same file system page cache