RAM issue

[IdavalapatiRamanjaneyulu]

When I use plyvel DB the RAM usage is keep on increasing.

Steps to reproduce: create.py # To create dummy data

import plyvel
import pickle

path = '/tmp/data'

p = plyvel.DB(path, create_if_missing=True)

for i in range(70000000):
    k = repr(i).encode('utf8')
    v = pickle.dumps('randomstring{}'.format(str(i)))
    p.put(k, v)

p.close()

get.py # To get the dummy data

  1 import random
  2 import gc
  3 import pickle
  4 
  5 import plyvel
  6 
  7 import pdb; pdb.set_trace()
  8 p = plyvel.DB('/tmp/data', create_if_missing=True)
  9 
 10 k = pickle.dumps(None)
 11 indices = random.sample(range(700000000), 1000000)
 12 indices = [repr(i).encode('utf8') for i in indices]
 13 
 14 import pdb; pdb.set_trace()
 15 obj = [pickle.loads(p.get(i, k)) for i in indices]
 16 del indices
 17 del obj
 18 gc.collect()
 19 
 20 p.close()
 21 gc.collect()
 22 import pdb; pdb.set_trace()

When I run python get.py I have observed the followed things in htop.

• At 7th line the stats are The process is using 16312(RES) - 9136(SHR) = 7176 KB of RAM at initial stage.
• At 14th line the stats are The variable indices is using 72776(RES) - 16312(previous RES) = 56464 KB of RAM
• After executing 15th line the stats are The variable obj took 776 MB(RES) - 673 MB(SHR) - 56464 KB(previous RAM usage) = 46.536 MB of RAM
• After executing 16th line the stats are The RAM freed after deleting indices is 776 - 730 = 40 MB
• After executing 17th line the stats are The RAM freed after deleting obj is 730 - 723 = 7 MB
• After executing 20th line the stats are Finally 61316 - 9756 = 51.560 MB Is left over.

Why the 50MB is left over in the RAM? And I am thinking the plyvel is caching the results (After closing only maximum RAM is getting released p.close()). Is there any mechanism to not cache any results?

[wbolster]

And I am thinking the plyvel is caching the results (After closing only maximum RAM is getting released p.close()). Is there any mechanism to not cache any results?

what makes you think that plyvel is caching anything? since plyvel does not contain any caching code.

leveldb itself does use some caches, and plyvel provides api (DB constructor args) to influence those. for example,. from leveldb.options.h:

  // If non-NULL, use the specified cache for blocks.
  // If NULL, leveldb will automatically create and use an 8MB internal cache.
  // Default: NULL
  Cache* block_cache;

[wbolster]

closing due to inactivity