search

westerndigitalcorporation / zenfs

ZenFS is a storage backend for RocksDB that enables support for ZNS SSDs and SMR HDDs.
GNU General Public License v2.0
243 stars 88 forks source link

running basic db_bench performance test results in an OOM #224

Closed ryderhsu0129 closed 2 years ago

ryderhsu0129 commented 2 years ago

Hi,

a basic db_bench performance testing we are running is causing our machines to run out of memory and get killed by the kernel.

[Test Log]

long_performance/0000_fillrandom.sh: line 7: 109458 Killed $TOOLS_DIR/db_bench $DB_BENCH_PARAMS >> $TEST_OUT

[dmesg]

Out of memory: Killed process 109458 (db_bench) total-vm:16862280kB, anon-rss:14872260kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:32036kB oom_score_adj:0

is there any advice to fix the issue?

yhr commented 2 years ago

@ryderhsu0129 : Could you share the parameters you were invoking the test with?

ryderhsu0129 commented 2 years ago

@yhr I just run the script zenfs_base_performance.sh, the parameters carried by the db_bench should be set by the script?

yhr commented 2 years ago

@ryderhsu0129 : right, if you are running the zenfs_base_performance.sh scripts the parameters are calculated by get_good_db_bench_params_for_zenfs.sh . Maybe, If you are testing on a device with huge zones(or one that reports wierd zone capacity values), the target file size base parameter calculation might cause the OOM. Hard to know for sure without knowing what you are testing on. As a reference, with zone capacity of ~1G, i get a memory usage of ~2G.

ryderhsu0129 commented 2 years ago

@yhr I have increased the memory capacity to 64GB, the capacity should be enough to finish the test. It has been tested for more than 24 hours, is this normal in general?If so how long does this test generally take? thanks.

yhr commented 2 years ago

@ryderhsu0129 : When I run it it normally takes about 8h or so, but it depends on the disk properties (write throughput, number of active zones)

ryderhsu0129 commented 2 years ago

@yhr OK, thanks for your reply.