deep011
commented
7 years ago You can set a small threads and mbuf_size=128(this need your all keys' length less than 128) to reduce the rmt memory usage. In the near future, i will add a config option to limit the memory usage. But this feature may change a lot of code, so the workload is not small.
Here is the sources memory data:
used_memory:2001297968 used_memory_human:1.86G used_memory_rss:2063167488 used_memory_rss_human:1.92G used_memory_peak:2003647128 used_memory_peak_human:1.87G total_system_memory:67426734080 total_system_memory_human:62.80G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:21474836480 maxmemory_human:20.00G maxmemory_policy:volatile-lru
1.86Gb data is stored per nodes(in productive environment, data size may over 20Gb), and there are three master nodes in my cluster. When migrating data, rmt has asked for up to 8G memory moving one node when finish dumping rdb from master and starting to redo commons to target cluster. So why using such a huge amount of memory? Or do you have any recommendation about moving a huge redis cluster?
Thanks a lot.
here is my watchdog for memory:
!/bin/bash
while true;do var=$(ps -e -o 'pid,comm,rsz'|grep redis-migrate) echo $var sleep 1 echo "=========================================" done;
and here is some part of data collecting when migrating.
4315 redis-migrate-t 2299276 4315 redis-migrate-t 2537820 4315 redis-migrate-t 2773684 4315 redis-migrate-t 2977248 4315 redis-migrate-t 3177380 4315 redis-migrate-t 3379796 4315 redis-migrate-t 3582552 4315 redis-migrate-t 3786856 4315 redis-migrate-t 3992220 4315 redis-migrate-t 4198120 4315 redis-migrate-t 4405084 4315 redis-migrate-t 4643704 4315 redis-migrate-t 4868276 4315 redis-migrate-t 5082396 4315 redis-migrate-t 5297076 4315 redis-migrate-t 5513200 4315 redis-migrate-t 5736984 4315 redis-migrate-t 6021188 4315 redis-migrate-t 6274092 4315 redis-migrate-t 6524820 4315 redis-migrate-t 6774952 4315 redis-migrate-t 7023940 4315 redis-migrate-t 7252052 4315 redis-migrate-t 7464704 4315 redis-migrate-t 7675420 4315 redis-migrate-t 7887032 4315 redis-migrate-t 8099404 4315 redis-migrate-t 8312348 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656 4315 redis-migrate-t 8476656