knizhnik / imcs

In-Memory Columnar Store extension for PostgreSQL
Apache License 2.0
203 stars 33 forks source link

the same data set consume double memory on head than 64fb76fbfb68285c35802b33cdf3da99a293fb95 #38

Closed amutu closed 10 years ago

amutu commented 10 years ago

on head(version 1.0.5),the cs used ~8GB while on 64fb76fbfb68285c35802b33cdf3da99a293fb95 used ~3.2GB,may be memory leak?

select crashlog_load(filter := $$logtime >= '2014-04-11' and logtime < '2014-04-12'$$);

crashlog_load

  30998550

(1 row)

select cs_used_memory();

cs_used_memory

 7971627008

(1 row)

on commit 64fb76fbfb68285c35802b33cdf3da99a293fb95: select crashlog_load(filter := $$logtime >= '2014-04-11' and logtime < '2014-04-12'$$);

crashlog_load

  30998550

(1 row)

select cs_used_memory();

cs_used_memory

 3209568256

(1 row)

imcs configuration: imcs.trace = on imcs.shmem_size = 40960 imcs.substitute_nulls = on imcs.autoload = off imcs.dictionary_size = 0

knizhnik commented 10 years ago

May be it is because of switched off RLE compression? In previous version it was enabled by default.

amutu commented 10 years ago

it looks like the imcs.use_rle = on in the postgresql.conf do not take effect: => show imcs.use_rle;

imcs.use_rle

on (1 row)

weixin_online_client_crash=> set work_mem to '10GB'; SET => select crashlog_load(filter := $$logtime >= '2014-04-11' and logtime < '2014-04-12'$$);

crashlog_load

  30998550

(1 row)

=> select cs_used_memory();

cs_used_memory

 7971627008

(1 row)

amutu commented 10 years ago

if I change the code: bool imcs_use_rle = false; to bool imcs_use_rle = true;

and set imcs.use_rle=on in postgresql.conf the memory used reduce to 3GB

if I comment the imcs.use_rle=on in postgresql.conf,then => show imcs.use_rle;
ERROR: unrecognized configuration parameter "imcs.use_rle"

knizhnik commented 10 years ago

Sorry, imcs.use_rle option was occasionally placed inside #ifdef IMCS_DISK_SUPPORT branch. Fixed.