core: support unix socket to zdb

maxux commented 2 years ago

Overall performance are increased by using unix socket to zdb

mohamedamer453 commented 2 years ago

I've tried verifying this but i kept facing some issues.

Firstly, i had 0-db started locally in sequential mode.

mohamed@mohamed-Inspiron-3576:~$ ./zdb --mode seq
[       0.000066][*] 0-db engine, v2.0.4 (commit v2.0.4)
[       0.000189][*] 0-db server, v2.0.4 (commit v2.0.4)
[       0.000199][+] system: acceleration engine: sse42
[       0.000224][+] system: running mode: sequential keys
[       0.000233][+] system: data chunk size: 256.00 MB
[       0.000250][+] system: instance id: 819321792
[       0.000260][+] system: setting up environments
[       0.000320][+] bootstrap: checking lockfile: ./zdb-index/.lockfile
[       0.000358][+] bootstrap: checking lockfile: ./zdb-data/.lockfile
[       0.000384][+] namespaces: pre-initializing
[       0.000397][+] namespaces: initializing
[       0.000405][+] namespaces: loading 'default'
[       0.000420][+] namespaces: checking index [./zdb-index/default]
[       0.000439][+] namespaces: checking data [./zdb-data/default]
[       0.000507][+] namespace: [default] opened, analyzing...
[       0.000520][+] -> maxsize: 0 (0.00 MB)
[       0.000530][+] -> password protection: no
[       0.000538][+] -> public access: yes
[       0.000550][+] -> worm mode: no
[       0.000565][+] index: initializing
[       0.000587][+] index: allocating dirty length: 1 bytes
[       0.000601][+] index loader: allocating sequential buffer map
[       0.000619][+] index: loader: fresh database created in sequential mode
[       0.000629][+] index: loader: initializing default seqmap
[       0.000641][+] index seq: mapping id 0 to file 0
[       0.000694][+] index: loading file: ./zdb-index/default/i0
[       0.000732][+] index: running mode: sequential keys
[       0.000751][+] index: writing 29 bytes on fd 5
[       0.000784][+] index: created at: 2022-08-30 09:34:50
[       0.000905][+] index: last open: 2022-08-30 09:44:07
[       0.000931][+] index: populating: ./zdb-index/default/i0
[       0.000942][+] index: loading in memory file: 0.00 MB
[       0.000958][+] index: last offset: 0
[       0.000970][+] index: loader: index is healthy
[       0.001013][+] index: active file: ./zdb-index/default/i0
[       0.001025][+] index: load: 0 entries
[       0.001038][+] index: datasize: 0.00 MB (0 bytes)
[       0.001050][+] index: raw usage: 0.0 KB (0 bytes)
[       0.001063][+] index seq: seqid 0 -> file 0
[       0.001091][+] data: writing 28 bytes to fd 6
[       0.001121][+] data: wrote 28 bytes to fd 6
[       0.001146][+] data: reading file, finding last entry
[       0.001164][+] data: entries read: 0, last offset: 0
[       0.001175][+] data: active file: ./zdb-data/default/d0
[       0.001232][+] namespaces: extra found: zdbfs-temp
[       0.001244][+] namespaces: loading 'zdbfs-temp'
[       0.001256][+] namespaces: checking index [./zdb-index/zdbfs-temp]
[       0.001273][+] namespaces: checking data [./zdb-data/zdbfs-temp]
[       0.001315][+] namespace: [zdbfs-temp] opened, analyzing...
[       0.001327][+] -> maxsize: 0 (0.00 MB)
[       0.001338][+] -> password protection: yes
[       0.001345][+] -> public access: no
[       0.001354][+] -> worm mode: no
[       0.001365][+] index: initializing
[       0.001385][+] index: allocating dirty length: 1 bytes
[       0.001398][+] index loader: allocating sequential buffer map
[       0.001415][+] index: loader: fresh database created in sequential mode
[       0.001427][+] index: loader: initializing default seqmap
[       0.001439][+] index seq: mapping id 0 to file 0
[       0.001476][+] index: loading file: ./zdb-index/zdbfs-temp/i0
[       0.001506][+] index: running mode: sequential keys
[       0.001520][+] index: writing 29 bytes on fd 8
[       0.001540][+] index: created at: 2022-08-30 09:38:16
[       0.001568][+] index: last open: 2022-08-30 09:44:07
[       0.001588][+] index: populating: ./zdb-index/zdbfs-temp/i0
[       0.001599][+] index: loading in memory file: 0.00 MB
[       0.001615][+] index seq: mapping id 0 to file 0
[       0.001627][+] index: last offset: 29
[       0.001641][+] index: loader: index is healthy
[       0.001663][+] index: active file: ./zdb-index/zdbfs-temp/i0
[       0.001674][+] index: load: 1 entries
[       0.001685][+] index: datasize: 0.00 MB (24 bytes)
[       0.001698][+] index: raw usage: 0.1 KB (72 bytes)
[       0.001710][+] index seq: seqid 0 -> file 0
[       0.001722][+] index seq: seqid 0 -> file 0
[       0.001749][+] data: writing 28 bytes to fd 9
[       0.001776][+] data: wrote 28 bytes to fd 9
[       0.001800][+] data: reading file, finding last entry
[       0.001824][+] data: entries read: 1, last offset: 28
[       0.001835][+] data: active file: ./zdb-data/zdbfs-temp/d0
[       0.001846][+] namespace: allocating new slot
[       0.001859][+] namespaces: extra found: zdbfs-meta
[       0.001869][+] namespaces: loading 'zdbfs-meta'
[       0.001881][+] namespaces: checking index [./zdb-index/zdbfs-meta]
[       0.001895][+] namespaces: checking data [./zdb-data/zdbfs-meta]
[       0.001928][+] namespace: [zdbfs-meta] opened, analyzing...
[       0.001940][+] -> maxsize: 0 (0.00 MB)
[       0.001949][+] -> password protection: no
[       0.001959][+] -> public access: yes
[       0.001970][+] -> worm mode: no
[       0.001983][+] index: initializing
[       0.002000][+] index: allocating dirty length: 1 bytes
[       0.002012][+] index loader: allocating sequential buffer map
[       0.002028][+] index: loader: fresh database created in sequential mode
[       0.002039][+] index: loader: initializing default seqmap
[       0.002049][+] index seq: mapping id 0 to file 0
[       0.002082][+] index: loading file: ./zdb-index/zdbfs-meta/i0
[       0.002109][+] index: running mode: sequential keys
[       0.002123][+] index: writing 29 bytes on fd 10
[       0.002141][+] index: created at: 2022-08-30 09:38:16
[       0.002167][+] index: last open: 2022-08-30 09:44:07
[       0.002186][+] index: populating: ./zdb-index/zdbfs-meta/i0
[       0.002197][+] index: loading in memory file: 0.00 MB
[       0.002209][+] index seq: mapping id 0 to file 0
[       0.002221][+] index: last offset: 281
[       0.002234][+] index: loader: index is healthy
[       0.002254][+] index: active file: ./zdb-index/zdbfs-meta/i0
[       0.002265][+] index: load: 4 entries
[       0.002275][+] index: datasize: 0.00 MB (1149 bytes)
[       0.002287][+] index: raw usage: 0.3 KB (288 bytes)
[       0.002297][+] index seq: seqid 0 -> file 0
[       0.002307][+] index seq: seqid 0 -> file 0
[       0.002339][+] data: writing 28 bytes to fd 11
[       0.002361][+] data: wrote 28 bytes to fd 11
[       0.002384][+] data: reading file, finding last entry
[       0.002441][+] data: entries read: 7, last offset: 268
[       0.002451][+] data: active file: ./zdb-data/zdbfs-meta/d0
[       0.002462][+] namespace: allocating new slot
[       0.002475][+] namespaces: extra found: zdbfs-data
[       0.002485][+] namespaces: loading 'zdbfs-data'
[       0.002496][+] namespaces: checking index [./zdb-index/zdbfs-data]
[       0.002511][+] namespaces: checking data [./zdb-data/zdbfs-data]
[       0.002541][+] namespace: [zdbfs-data] opened, analyzing...
[       0.002552][+] -> maxsize: 0 (0.00 MB)
[       0.002564][+] -> password protection: no
[       0.002573][+] -> public access: yes
[       0.002585][+] -> worm mode: no
[       0.002599][+] index: initializing
[       0.002617][+] index: allocating dirty length: 1 bytes
[       0.002628][+] index loader: allocating sequential buffer map
[       0.002652][+] index: loader: fresh database created in sequential mode
[       0.002662][+] index: loader: initializing default seqmap
[       0.002672][+] index seq: mapping id 0 to file 0
[       0.002703][+] index: loading file: ./zdb-index/zdbfs-data/i0
[       0.002731][+] index: running mode: sequential keys
[       0.002744][+] index: writing 29 bytes on fd 12
[       0.002763][+] index: created at: 2022-08-30 09:38:16
[       0.002788][+] index: last open: 2022-08-30 09:44:07
[       0.002805][+] index: populating: ./zdb-index/zdbfs-data/i0
[       0.002818][+] index: loading in memory file: 0.00 MB
[       0.002835][+] index seq: mapping id 0 to file 0
[       0.002846][+] index: last offset: 29
[       0.002860][+] index: loader: index is healthy
[       0.002879][+] index: active file: ./zdb-index/zdbfs-data/i0
[       0.002890][+] index: load: 1 entries
[       0.002899][+] index: datasize: 0.00 MB (24 bytes)
[       0.002910][+] index: raw usage: 0.1 KB (72 bytes)
[       0.002922][+] index seq: seqid 0 -> file 0
[       0.002931][+] index seq: seqid 0 -> file 0
[       0.002958][+] data: writing 28 bytes to fd 13
[       0.002980][+] data: wrote 28 bytes to fd 13
[       0.003004][+] data: reading file, finding last entry
[       0.003027][+] data: entries read: 1, last offset: 28
[       0.003038][+] data: active file: ./zdb-data/zdbfs-data/d0
[       0.003048][+] namespace: allocating new slot
[       0.003086][+] namespaces: 3 extra namespaces loaded
[       0.003092][+] network: looking for: host: ::, port: 9900
[       0.003115][+] listen request: tcp://:::9900 (fd: 7)
[       0.003125][+] listening on socket 7

but when i tried to do the next that's mentioned in the readme i kept getting an error.

mohamed@mohamed-Inspiron-3576:~$ ./zdbfs -o autons /mnt/zdbfs
[+] initializing zdbfs v0.1.10
fuse: bad mount point `/mnt/zdbfs': No such file or directory

also i found from the available options that there's a unix option that can be added after -o and that's what i'm supposed to verify in this issue but i really didn't understand what i'm supposed to pass to this option.

i feel like there's a couple of steps that i'm missing or doing wrong in this flow, it would be great if you can fill me in on how to try this properly with the unix option to be able to verify this issue.

maxux commented 2 years ago

The problem you face is the missing /mnt/zdbfs directory: mkdir /mnt/zdbfs (you probably need to sudo it). Just this should fix your main issue.

But, in order to try unix socket, you need extra things. By default zdb listen on 127.0.0.1 on tcp, not unix socket. You need to run zdb with --socket /tmp/zdb.sock for example.

Then you can ask zdbfs to connect zdb via unix socket: ./zdbfs -o unix=/tmp/zdb.sock ...

mohamedamer453 commented 2 years ago

The problem you face is the missing /mnt/zdbfs directory: mkdir /mnt/zdbfs (you probably need to sudo it). Just this should fix your main issue.

indeed after creating this directory the original issue was fixed, but now i'm getting a different error. here are the steps i followed:

Firstly i ran zdb with the --socket /tmp/zdb.sock option that you mentioned and it worked and from what i understood from the logs it was listening on unix socket . [ 0.000768][+] listen request: unix:///tmp/zdb.sock (fd: 7) and i tried running it using sudo to avoid any permission issues (i tried it also without sudo and got the same error in zdbfs.)

mohamed@mohamed-Inspiron-3576:~$ sudo ./zdb --socket /tmp/zdb.sock
[sudo] password for mohamed: 
[       0.000016][*] 0-db engine, v2.0.4 (commit v2.0.4)
[       0.000045][*] 0-db server, v2.0.4 (commit v2.0.4)
[       0.000047][+] system: acceleration engine: sse42
[       0.000051][+] system: running mode: mixed mode
[       0.000053][+] system: data chunk size: 256.00 MB
[       0.000055][+] system: instance id: 667386063
[       0.000057][+] system: setting up environments
[       0.000070][+] bootstrap: checking lockfile: ./zdb-index/.lockfile
[       0.000077][+] bootstrap: checking lockfile: ./zdb-data/.lockfile
[       0.000087][+] namespaces: pre-initializing
[       0.000090][+] namespaces: initializing
[       0.000091][+] namespaces: pre-allocating index (16777216 lazy branches)
[       0.000098][+] namespaces: loading 'default'
[       0.000103][+] namespaces: checking index [./zdb-index/default]
[       0.000107][+] namespaces: checking data [./zdb-data/default]
[       0.000115][+] namespace: [default] opened, analyzing...
[       0.000118][+] -> maxsize: 0 (0.00 MB)
[       0.000121][+] -> password protection: no
[       0.000123][+] -> public access: yes
[       0.000124][+] -> worm mode: no
[       0.000128][+] index: initializing
[       0.000133][+] index: allocating dirty length: 1 bytes
[       0.000142][+] index: loading file: ./zdb-index/default/i0
[       0.000153][+] index: running mode: sequential keys
[       0.000156][+] index loader: allocating sequential buffer map
[       0.000160][+] index: loader: fresh database created in sequential mode
[       0.000163][+] index: loader: initializing default seqmap
[       0.000165][+] index seq: mapping id 0 to file 0
[       0.000169][+] index: writing 29 bytes on fd 5
[       0.000175][+] index: created at: 2022-08-30 09:34:50
[       0.000207][+] index: last open: 2022-09-01 11:36:16
[       0.000214][+] index: populating: ./zdb-index/default/i0
[       0.000217][+] index: loading in memory file: 0.00 MB
[       0.000223][+] index: last offset: 0
[       0.000227][+] index: loader: index is healthy
[       0.000233][+] index: active file: ./zdb-index/default/i0
[       0.000236][+] index: load: 0 entries
[       0.000238][+] index: datasize: 0.00 MB (0 bytes)
[       0.000241][+] index: raw usage: 0.0 KB (0 bytes)
[       0.000244][+] index seq: seqid 0 -> file 0
[       0.000251][+] data: writing 28 bytes to fd 6
[       0.000258][+] data: wrote 28 bytes to fd 6
[       0.000264][+] data: reading file, finding last entry
[       0.000269][+] data: entries read: 0, last offset: 0
[       0.000271][+] data: active file: ./zdb-data/default/d0
[       0.000285][+] namespaces: extra found: zdbfs-temp
[       0.000287][+] namespaces: loading 'zdbfs-temp'
[       0.000291][+] namespaces: checking index [./zdb-index/zdbfs-temp]
[       0.000295][+] namespaces: checking data [./zdb-data/zdbfs-temp]
[       0.000308][+] namespace: [zdbfs-temp] opened, analyzing...
[       0.000311][+] -> maxsize: 0 (0.00 MB)
[       0.000313][+] -> password protection: yes
[       0.000316][+] -> public access: no
[       0.000318][+] -> worm mode: no
[       0.000321][+] index: initializing
[       0.000325][+] index: allocating dirty length: 1 bytes
[       0.000333][+] index: loading file: ./zdb-index/zdbfs-temp/i0
[       0.000341][+] index: running mode: sequential keys
[       0.000344][+] index loader: allocating sequential buffer map
[       0.000348][+] index: loader: fresh database created in sequential mode
[       0.000350][+] index: loader: initializing default seqmap
[       0.000353][+] index seq: mapping id 0 to file 0
[       0.000355][+] index: writing 29 bytes on fd 8
[       0.000360][+] index: created at: 2022-08-30 09:38:16
[       0.000367][+] index: last open: 2022-09-01 11:36:16
[       0.000371][+] index: populating: ./zdb-index/zdbfs-temp/i0
[       0.000374][+] index: loading in memory file: 0.00 MB
[       0.000378][+] index seq: mapping id 0 to file 0
[       0.000381][+] index: last offset: 29
[       0.000384][+] index: loader: index is healthy
[       0.000389][+] index: active file: ./zdb-index/zdbfs-temp/i0
[       0.000392][+] index: load: 1 entries
[       0.000394][+] index: datasize: 0.00 MB (24 bytes)
[       0.000397][+] index: raw usage: 0.1 KB (72 bytes)
[       0.000400][+] index seq: seqid 0 -> file 0
[       0.000402][+] index seq: seqid 0 -> file 0
[       0.000409][+] data: writing 28 bytes to fd 9
[       0.000415][+] data: wrote 28 bytes to fd 9
[       0.000421][+] data: reading file, finding last entry
[       0.000427][+] data: entries read: 1, last offset: 28
[       0.000429][+] data: active file: ./zdb-data/zdbfs-temp/d0
[       0.000432][+] namespace: allocating new slot
[       0.000435][+] namespaces: extra found: zdbfs-meta
[       0.000437][+] namespaces: loading 'zdbfs-meta'
[       0.000440][+] namespaces: checking index [./zdb-index/zdbfs-meta]
[       0.000445][+] namespaces: checking data [./zdb-data/zdbfs-meta]
[       0.000452][+] namespace: [zdbfs-meta] opened, analyzing...
[       0.000455][+] -> maxsize: 0 (0.00 MB)
[       0.000457][+] -> password protection: no
[       0.000459][+] -> public access: yes
[       0.000462][+] -> worm mode: no
[       0.000465][+] index: initializing
[       0.000469][+] index: allocating dirty length: 1 bytes
[       0.000476][+] index: loading file: ./zdb-index/zdbfs-meta/i0
[       0.000483][+] index: running mode: sequential keys
[       0.000486][+] index loader: allocating sequential buffer map
[       0.000490][+] index: loader: fresh database created in sequential mode
[       0.000492][+] index: loader: initializing default seqmap
[       0.000494][+] index seq: mapping id 0 to file 0
[       0.000497][+] index: writing 29 bytes on fd 10
[       0.000502][+] index: created at: 2022-08-30 09:38:16
[       0.000508][+] index: last open: 2022-09-01 11:36:16
[       0.000512][+] index: populating: ./zdb-index/zdbfs-meta/i0
[       0.000515][+] index: loading in memory file: 0.00 MB
[       0.000519][+] index seq: mapping id 0 to file 0
[       0.000522][+] index: last offset: 281
[       0.000525][+] index: loader: index is healthy
[       0.000530][+] index: active file: ./zdb-index/zdbfs-meta/i0
[       0.000533][+] index: load: 4 entries
[       0.000535][+] index: datasize: 0.00 MB (1149 bytes)
[       0.000538][+] index: raw usage: 0.3 KB (288 bytes)
[       0.000540][+] index seq: seqid 0 -> file 0
[       0.000543][+] index seq: seqid 0 -> file 0
[       0.000551][+] data: writing 28 bytes to fd 11
[       0.000556][+] data: wrote 28 bytes to fd 11
[       0.000562][+] data: reading file, finding last entry
[       0.000576][+] data: entries read: 7, last offset: 268
[       0.000579][+] data: active file: ./zdb-data/zdbfs-meta/d0
[       0.000581][+] namespace: allocating new slot
[       0.000583][+] namespaces: extra found: zdbfs-data
[       0.000586][+] namespaces: loading 'zdbfs-data'
[       0.000589][+] namespaces: checking index [./zdb-index/zdbfs-data]
[       0.000593][+] namespaces: checking data [./zdb-data/zdbfs-data]
[       0.000600][+] namespace: [zdbfs-data] opened, analyzing...
[       0.000603][+] -> maxsize: 0 (0.00 MB)
[       0.000605][+] -> password protection: no
[       0.000607][+] -> public access: yes
[       0.000610][+] -> worm mode: no
[       0.000613][+] index: initializing
[       0.000617][+] index: allocating dirty length: 1 bytes
[       0.000624][+] index: loading file: ./zdb-index/zdbfs-data/i0
[       0.000631][+] index: running mode: sequential keys
[       0.000633][+] index loader: allocating sequential buffer map
[       0.000639][+] index: loader: fresh database created in sequential mode
[       0.000642][+] index: loader: initializing default seqmap
[       0.000644][+] index seq: mapping id 0 to file 0
[       0.000647][+] index: writing 29 bytes on fd 12
[       0.000652][+] index: created at: 2022-08-30 09:38:16
[       0.000657][+] index: last open: 2022-09-01 11:36:16
[       0.000662][+] index: populating: ./zdb-index/zdbfs-data/i0
[       0.000664][+] index: loading in memory file: 0.00 MB
[       0.000668][+] index seq: mapping id 0 to file 0
[       0.000670][+] index: last offset: 29
[       0.000674][+] index: loader: index is healthy
[       0.000679][+] index: active file: ./zdb-index/zdbfs-data/i0
[       0.000681][+] index: load: 1 entries
[       0.000683][+] index: datasize: 0.00 MB (24 bytes)
[       0.000686][+] index: raw usage: 0.1 KB (72 bytes)
[       0.000688][+] index seq: seqid 0 -> file 0
[       0.000691][+] index seq: seqid 0 -> file 0
[       0.000698][+] data: writing 28 bytes to fd 13
[       0.000703][+] data: wrote 28 bytes to fd 13
[       0.000709][+] data: reading file, finding last entry
[       0.000714][+] data: entries read: 1, last offset: 28
[       0.000717][+] data: active file: ./zdb-data/zdbfs-data/d0
[       0.000719][+] namespace: allocating new slot
[       0.000724][+] namespaces: 3 extra namespaces loaded
[       0.000768][+] listen request: unix:///tmp/zdb.sock (fd: 7)
[       0.000776][+] listening on socket 7

Then i started zdbfs with the unix option in the newly created directory /mnt/zdbfs using the mentioned command ./zdbfs -o unix=/tmp/zdb.sock /mnt/zdbfs but i got the following error [-] zdb: connect: [localhost:9900]: Connection refused

maxux commented 2 years ago

Are you using latest release or latest commit ? Use master branch to latest version to test.

mohamedamer453 commented 2 years ago

for 0-db i cloned the repo and followed the build instructions and i'm using the zdb binary from the /bin directory.

and for 0-dbfs i downloaded the latest binary from here because when i tried to build it locally after cloning the repo i got the following error.

mohamed@mohamed-Inspiron-3576:~/Desktop/Github/zdbfs/0-db-fs$ make 
make -C src all
make[1]: Entering directory '/home/mohamed/Desktop/Github/zdbfs/0-db-fs/src'
cc -g -std=gnu99 -O0 -W -Wall -Wextra -Wno-implicit-fallthrough -I/usr/include/fuse3 -c cache.c
cache.c:7:10: fatal error: fuse_lowlevel.h: No such file or directory
    7 | #include <fuse_lowlevel.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:25: cache.o] Error 1
make[1]: Leaving directory '/home/mohamed/Desktop/Github/zdbfs/0-db-fs/src'
make: *** [Makefile:2: all] Error 2

maxux commented 2 years ago

You need to install fuse devel files (apt-get install libfuse-dev probably). Latest release doesn't support unix socket yet so you need to compile it.