DBMS Memory Usage Always Increases

apavlo commented 7 years ago

The DBMS process keeps growing in memory even if the logical database size does not increase.

For this test, I first loaded a 1GB YCSB database (scalefactor=1000) with OLTP-Bench. This was the size of the peloton process in htop before I started executing txns (yes I know there are better ways to get this measurement):

VIRT: 3708M
RES:  2227M

I then had a single worker thread execute only the UpdateRecord transaction for 20 minutes. The throughput was 11440 txn/sec. Afterwards, the DBMS size was this:

VIRT: 22.9G
RES:  21.4G

This should not happen since UpdateRecord never inserts or deletes a tuple. To make sure that the database was the same size, I ran a simple count query and it matched what should be there:

postgres=# SELECT COUNT(*) FROM USERTABLE;
 count(*) 
----------
  1000000
(1 row)

Something is obviously not right.

apavlo commented 7 years ago

@yingjunwu - Latest version is still not reclaiming memory. CPU is not spiked at 100% anymore. Tested with 1fc29ad10ac86b7abaf3493a9ed24122e596189b

BEFORE

VIRT: 3628M
RES:  3140M

AFTER:

VIRT: 20.6G
RES:  20.2G

AFAIK, the GC is supposed to be turned on:

https://github.com/cmu-db/peloton/blob/master/src/gc/gc_manager_factory.cpp#L19

yingjunwu commented 7 years ago

I'll take a look at it.

On Fri, Jan 6, 2017 at 6:16 AM Andy Pavlo notifications@github.com wrote:

@yingjunwu https://github.com/yingjunwu - Latest version is still not reclaiming memory. CPU is not spiked at 100% anymore. Tested with 1fc29ad https://github.com/cmu-db/peloton/commit/1fc29ad10ac86b7abaf3493a9ed24122e596189b

BEFORE

VIRT: 3628M

RES: 3140M

*AFTER:**

VIRT: 20.6G

RES: 20.2G

AFAIK, the GC is supposed to be turned on:

https://github.com/cmu-db/peloton/blob/master/src/gc/gc_manager_factory.cpp#L19

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cmu-db/peloton/issues/442#issuecomment-270786131, or mute the thread https://github.com/notifications/unsubscribe-auth/APPCBoRqoFgvVck3V2hqHWOb4fhTO2g3ks5rPXmwgaJpZM4LXxvb .

cmu-db / peloton

DBMS Memory Usage Always Increases #442