search

dain / leveldb

Port of LevelDB to Java
Apache License 2.0
1.54k stars 429 forks source link

Large SST files (ignoring VersionSet.TARGET_FILE_SIZE?) #65

Closed lwhite1 closed 8 years ago

lwhite1 commented 8 years ago

In my databases, SSTable file sizes are more or less the same within an individual db. Varying either the content or db options, can cause sst files to be ~2MB in one database, ~23MB in another, ~39MB in a third, etc. That's the approximate size of every sst file in the system, in databases with dozens to hundreds of SST files.

Is there something I can do to control this? The large db files perform poorly on read-heavy loads.

thanks,

rnarubin commented 8 years ago

tables in levels 1+ conform to the TARGET_FILE_SIZE; tables written to level 0 will be around the writeBufferSize, which is configurable in Options

lwhite1 commented 8 years ago

All of the files are approximately the same size. They can't all be level 0, could they?

On Monday, December 14, 2015, rnarubin notifications@github.com wrote:

tables in levels 1+ conform to the TARGET_FILE_SIZE; tables written to level 0 will be around the writeBufferSize, which is configurable in Options

— Reply to this email directly or view it on GitHub https://github.com/dain/leveldb/issues/65#issuecomment-164524746.

rnarubin commented 8 years ago

how large are your keys/values? they don't get split up across tables; for example, if a table being built is 1MB and it's given a 4MB key-value pair, it'll grow to 5MB before checking the size breakpoint

lwhite1 commented 8 years ago

Values are around 1k

On Monday, December 14, 2015, rnarubin notifications@github.com wrote:

how large are your keys/values? they don't get split up across tables; for example, if a table being built is 1MB and it's given a 4MB key-value pair, it'll grow to 5MB before checking the size breakpoint

— Reply to this email directly or view it on GitHub https://github.com/dain/leveldb/issues/65#issuecomment-164527168.

rnarubin commented 8 years ago

incidentally, you can inspect the MANIFEST file using google's leveldbutil from their package. if you clone their repo, make leveldbutil creates a tool with which you can dump the contents of certain leveldb files. In this case, leveldbutil dump /path/to/MANIFEST-xxxxxx will list all live files, which for table files includes their level number. The output is a bit cryptic, but you'll have something like

"AddFile: 2 1510 1923066 '\xfc\xdf\xfb;Qs+\xbeK6\xb5\x1e\xf0..." the 2 here is the level, the 1510 is the file number, and the 1923066 is the file size in bytes. the hex string that i've cut off here is the beginning of the smallest key in the table, it'll also have the largest key

On Mon, Dec 14, 2015 at 1:21 PM, Larry White notifications@github.com wrote:

Values are around 1k

On Monday, December 14, 2015, rnarubin notifications@github.com wrote:

how large are your keys/values? they don't get split up across tables; for example, if a table being built is 1MB and it's given a 4MB key-value pair, it'll grow to 5MB before checking the size breakpoint

— Reply to this email directly or view it on GitHub https://github.com/dain/leveldb/issues/65#issuecomment-164527168.

— Reply to this email directly or view it on GitHub https://github.com/dain/leveldb/issues/65#issuecomment-164531992.