search

estraier / tkrzw

a set of implementations of DBM
Apache License 2.0
169 stars 21 forks source link

Make check fail #4

Closed CuriousTim closed 3 years ago

CuriousTim commented 3 years ago

Hello, I attempted to build tkrzw on my local machine and upon running make check (I ran ./configure and make first), I get the following error:

...
make check-hashdbm-perf
make[1]: Entering directory '/home/charles/Documents/code/tkrzw'
rm -Rf casket*
LD_LIBRARY_PATH=.:/usr/local/lib:  ./tkrzw_dbm_perf sequence --dbm hash --file mmap-para --path casket.tkh \
  --iter 20000 --threads 5 --size 8 --buckets 100000 --lock_mem_buckets
OpenAdvanced failed: SYSTEM_ERROR: mlock: no enough memory
Setting: impl=hash num_iterations=20000 value_size=8 num_threads=5
Set failed: PRECONDITION_ERROR: not opened database
Set failed: PRECONDITION_ERROR: not opened database
Set failed: Set failed: PRECONDITION_ERROR: not opened database
PRECONDITION_ERROR: not opened database
Set failed: PRECONDITION_ERROR: not opened database
Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database
done (elapsed=0.000009)
Setting done: elapsed_time=0.000317 num_records=-1 qps=315361203 mem=2596000
  file_size=-1 eff_data_size=-1 efficiency=100.00%
  num_buckets=-1 load_factor=1.00
Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data
Getting: impl=hash num_iterations=20000 value_size=8 num_threads=5
Get failed: PRECONDITION_ERROR: not opened database
Get failed: PRECONDITION_ERROR: not opened database
Get failed: PRECONDITION_ERROR: not opened database
Get failed: PRECONDITION_ERROR: not opened database
Get failed: PRECONDITION_ERROR: not opened database
Getting done: elapsed_time=0.000200 num_records=-1 qps=499917044 mem=2660000
  file_size=-1 eff_data_size=-1 efficiency=100.00%
  num_buckets=-1 load_factor=1.00
Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data
Removing: impl=hash num_iterations=20000 value_size=8 num_threads=5
Remove failed: PRECONDITION_ERROR: not opened database
Remove failed: PRECONDITION_ERROR: not opened database
Remove failed: PRECONDITION_ERROR: not opened database
Remove failed: PRECONDITION_ERROR: not opened database
Remove failed: PRECONDITION_ERROR: not opened database
Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database
done (elapsed=0.000015)
Removing done: elapsed_time=0.000281 num_records=-1 qps=355750975 mem=2660000
  file_size=-1 eff_data_size=-1 efficiency=100.00%
  num_buckets=-1 load_factor=1.00
Close failed: PRECONDITION_ERROR: not opened database
...

Can this error be ignored?

estraier commented 3 years ago

it says "no enough memory". How much is RAM available on your system?

On Sat, Dec 19, 2020, 04:48 CuriousTim notifications@github.com wrote:

Hello, I attempted to build tkrzw on my local machine and upon running make check (I ran ./configure and make first), I get the following error:

... make check-hashdbm-perf make[1]: Entering directory '/home/charles/Documents/code/tkrzw' rm -Rf casket* LD_LIBRARY_PATH=.:/usr/local/lib: ./tkrzw_dbm_perf sequence --dbm hash --file mmap-para --path casket.tkh \ --iter 20000 --threads 5 --size 8 --buckets 100000 --lock_mem_buckets OpenAdvanced failed: SYSTEM_ERROR: mlock: no enough memory Setting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Set failed: PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Set failed: Set failed: PRECONDITION_ERROR: not opened database PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000009) Setting done: elapsed_time=0.000317 num_records=-1 qps=315361203 mem=2596000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Getting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Getting done: elapsed_time=0.000200 num_records=-1 qps=499917044 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Removing: impl=hash num_iterations=20000 value_size=8 num_threads=5 Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000015) Removing done: elapsed_time=0.000281 num_records=-1 qps=355750975 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database ...

Can this error be ignored?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGJVRB5IIFV2PXQYY4ATQLSVOWX5ANCNFSM4VBTOP3A .

CuriousTim commented 3 years ago

My system has 16GB RAM which I figured would be enough. No good?

2020년 12월 18일 (금) 17:58, Mikio Hirabayashi notifications@github.com님이 작성:

it says "no enough memory". How much is RAM available on your system?

On Sat, Dec 19, 2020, 04:48 CuriousTim notifications@github.com wrote:

Hello, I attempted to build tkrzw on my local machine and upon running make check (I ran ./configure and make first), I get the following error:

... make check-hashdbm-perf make[1]: Entering directory '/home/charles/Documents/code/tkrzw' rm -Rf casket* LD_LIBRARY_PATH=.:/usr/local/lib: ./tkrzw_dbm_perf sequence --dbm hash --file mmap-para --path casket.tkh \ --iter 20000 --threads 5 --size 8 --buckets 100000 --lock_mem_buckets OpenAdvanced failed: SYSTEM_ERROR: mlock: no enough memory Setting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Set failed: PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Set failed: Set failed: PRECONDITION_ERROR: not opened database PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000009) Setting done: elapsed_time=0.000317 num_records=-1 qps=315361203 mem=2596000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Getting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Getting done: elapsed_time=0.000200 num_records=-1 qps=499917044 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Removing: impl=hash num_iterations=20000 value_size=8 num_threads=5 Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000015) Removing done: elapsed_time=0.000281 num_records=-1 qps=355750975 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database ...

Can this error be ignored?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AQGJVRB5IIFV2PXQYY4ATQLSVOWX5ANCNFSM4VBTOP3A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#issuecomment-748359781, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM75BCGYBXWXM5EZTG3GO3SVPNANANCNFSM4VBTOP3A .

estraier commented 3 years ago

16GB must be enough as a system. Then, I guess "ulimit" of the user is not enough.

On Sat, Dec 19, 2020, 14:33 CuriousTim notifications@github.com wrote:

My system has 16GB RAM which I figured would be enough. No good?

2020년 12월 18일 (금) 17:58, Mikio Hirabayashi notifications@github.com님이 작성:

it says "no enough memory". How much is RAM available on your system?

On Sat, Dec 19, 2020, 04:48 CuriousTim notifications@github.com wrote:

Hello, I attempted to build tkrzw on my local machine and upon running make check (I ran ./configure and make first), I get the following error:

... make check-hashdbm-perf make[1]: Entering directory '/home/charles/Documents/code/tkrzw' rm -Rf casket* LD_LIBRARY_PATH=.:/usr/local/lib: ./tkrzw_dbm_perf sequence --dbm hash --file mmap-para --path casket.tkh \ --iter 20000 --threads 5 --size 8 --buckets 100000 --lock_mem_buckets OpenAdvanced failed: SYSTEM_ERROR: mlock: no enough memory Setting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Set failed: PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Set failed: Set failed: PRECONDITION_ERROR: not opened database PRECONDITION_ERROR: not opened database Set failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000009) Setting done: elapsed_time=0.000317 num_records=-1 qps=315361203 mem=2596000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Getting: impl=hash num_iterations=20000 value_size=8 num_threads=5 Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Get failed: PRECONDITION_ERROR: not opened database Getting done: elapsed_time=0.000200 num_records=-1 qps=499917044 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database

OpenAdvanced failed: BROKEN_DATA_ERROR: bad magic data Removing: impl=hash num_iterations=20000 value_size=8 num_threads=5 Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Remove failed: PRECONDITION_ERROR: not opened database Synchronizing: ... Synchronize failed: PRECONDITION_ERROR: not opened database done (elapsed=0.000015) Removing done: elapsed_time=0.000281 num_records=-1 qps=355750975 mem=2660000 file_size=-1 eff_data_size=-1 efficiency=100.00% num_buckets=-1 load_factor=1.00 Close failed: PRECONDITION_ERROR: not opened database ...

Can this error be ignored?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AQGJVRB5IIFV2PXQYY4ATQLSVOWX5ANCNFSM4VBTOP3A

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#issuecomment-748359781, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABM75BCGYBXWXM5EZTG3GO3SVPNANANCNFSM4VBTOP3A

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#issuecomment-748422789, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGJVRFSAPI3MOSCB6UKASDSVQ3JDANCNFSM4VBTOP3A .

CuriousTim commented 3 years ago

This is my personal machine so I didn't think there would be a user memory limit. I checked ulimit -v and I get unlimited. But if this test failing is just due to memory limits, is it possible to limit the memory usage of the library and still use it? I am mostly interested in the on-disk databases.

estraier commented 3 years ago

Actually, the failing test is for the memory locking featere, which is not activated with the file database by default. so, you can ignore the error.

btw, please do ulimit -l it shows the limit of memory locking, which causes thw error, I guess.

On Sun, Dec 20, 2020, 02:05 CuriousTim notifications@github.com wrote:

This is my personal machine so I didn't think there would be a user memory limit. I checked ulimit -v and I get unlimited. But if this test failing is just due to memory limits, is it possible to limit the memory usage of the library and still use it? I am mostly interested in the on-disk databases.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#issuecomment-748499060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGJVRACISH5I65YOSNZDIDSVTMODANCNFSM4VBTOP3A .

CuriousTim commented 3 years ago

That seems to be it.

$ ulimit -l
64

with size in KB. Just out of curiosity, would I need to set this limit to the size of the in-memory database to pass the check?

estraier commented 3 years ago

The header and the bucket sections only are locked. 100000 is the number of buckets thus, 4k + 400k is required. So, please set the limit 512k or something.

On Sun, Dec 20, 2020, 04:33 CuriousTim notifications@github.com wrote:

That seems to be it.

$ ulimit -l 64

with size in KB. Just out of curiosity, would I need to set this limit to the size of the in-memory database to pass the check?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#issuecomment-748515594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGJVRD6YWWRPC2K3GPMHQ3SVT5YTANCNFSM4VBTOP3A .

CuriousTim commented 3 years ago

Awesome, that does the trick. Thank you for your help.

estraier commented 3 years ago

Hi, The error seems to come from the user limit of memory locking. Please check it by this command. $ ulimit -l

You can loosen the limit by this, then please run "make check". $ ulimit -S -l 1024

On Mon, Dec 21, 2020 at 10:11 AM CuriousTim notifications@github.com wrote:

Closed #4 https://github.com/estraier/tkrzw/issues/4.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/estraier/tkrzw/issues/4#event-4133905574, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGJVRAW333UG4QY7HOT233SV2OFHANCNFSM4VBTOP3A .