Open ecsuae opened 9 months ago
@ecsuae
Can you try restoring (for now) your .bashrc
to default version while mainting source /etc/nhi/nhi.bash
, and then rebooting your computer?
Also where is the history of commands stored?
all information that nhi gathers is stored in /var/nhi/db
which is sqlite db.
Thanks for the reply, I have removed source /etc/nhi/nhi.bash from .bashrc and manually run source /etc/nhi/nhi.bash, I can see the db file exists ls -la /var/nhi/db -rwxrwxrwx 1 root root 8192 Sep 14 03:15 /var/nhi/db
Do I need to restart saver ?
also how can I see the command run? I tried nhi fetch {-1} no such shell session: 0
if you can guide how can I see previous commands ? I am logged in as root user.
Can you check if nhid daemon is running? (for example via htop by simply searching nhid)
Do I need to restart saver ?
No, but it would be the best option. For example, if nhid is broken and not running for some arbitrary reason, and then you start nhid manually, then nhid won't know about existence of shells sessions created before, but nhid will still receive information about commands that will be run, while not being able associate them with any shell session. It will lead to some small fails in nhid which will be written to /tmp/nhi.log.
nhid daemon is not running i think problem is with my installation.let me show you all my steps and their outcome
apt-get install binutils gawk sqlite3 libsqlite3-dev libbpf-dev -y mkdir nhi && cd nhi wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/install wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhi wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhi.bash wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhi.bpf.o wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhi.zsh wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhid wget https://github.com/strang1ato/nhi/releases/download/v0.2.6/nhid.service
root@kam:~/nhi# ls -la total 6067 drwxr-xr-x 2 root root 9 Sep 18 17:09 . drwx------ 5 root root 13 Sep 18 17:06 .. -rw-r--r-- 1 root root 601 Mar 19 2023 install -rw-r--r-- 1 root root 7803640 Mar 19 2023 nhi -rw-r--r-- 1 root root 1110 Mar 19 2023 nhi.bash -rw-r--r-- 1 root root 891464 Mar 19 2023 nhi.bpf.o -rw-r--r-- 1 root root 592 Mar 19 2023 nhi.zsh -rw-r--r-- 1 root root 32560 Mar 19 2023 nhid -rw-r--r-- 1 root root 148 Mar 19 2023 nhid.service
sudo -E bash ./install wal Created symlink /etc/systemd/system/multi-user.target.wants/nhid.service → /etc/systemd/system/nhid.service.
Rebooted
systemctl status nhid ● nhid.service - nhi daemon Loaded: loaded (/etc/systemd/system/nhid.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2023-09-18 17:11:50 UTC; 20s ago Process: 301 ExecStart=nhid (code=exited, status=127) Main PID: 301 (code=exited, status=127) CPU: 1ms
Sep 18 17:11:50 kam systemd[1]: nhid.service: Scheduled restart job, restart counter is at 5. Sep 18 17:11:50 kam systemd[1]: Stopped nhi daemon. Sep 18 17:11:50 kam systemd[1]: nhid.service: Start request repeated too quickly. Sep 18 17:11:50 kam systemd[1]: nhid.service: Failed with result 'exit-code'. Sep 18 17:11:50 kam systemd[1]: Failed to start nhi daemon.
root@kam:~# echo nhi test nhi test nhi fetch {-1} no such shell session: 0
Any suggestions what. I am doing wrong?
Can you post here your .bashrc
? Maybe some configuration there is breaking nhid
here's my .bashrc
root@kam:~# cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# Note: PS1 and umask are already set in /etc/profile. You should not
# need this unless you want different defaults for root.
# PS1='${debian_chroot:+($debian_chroot)}\h:\w\$ '
# umask 022
# You may uncomment the following lines if you want `ls' to be colorized:
# export LS_OPTIONS='--color=auto'
# eval "$(dircolors)"
# alias ls='ls $LS_OPTIONS'
# alias ll='ls $LS_OPTIONS -l'
# alias l='ls $LS_OPTIONS -lA'
#
# Some more alias to avoid making mistakes:
# alias rm='rm -i'
# alias cp='cp -i'
# alias mv='mv -i'
#sipsak_call() {
# sipsak -T -s sip:$1@stirshaken.corvitbpo.com;user=phone
#}
source /etc/nhi/nhi.bash
Maybe for some reason systemctl can't find nhid without full path.
Can you replace this line by ExecStart=/usr/bin/nhid
in /etc/systemd/system/nhid.service
And then run: systemctl enable nhid && systemctl start nhid
while keeping in mind this comment
I have tried it still service not starting. root@kam:~# systemctl status nhid ● nhid.service - nhi daemon Loaded: loaded (/etc/systemd/system/nhid.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2023-09-19 17:51:09 UTC; 9s ago Process: 1119 ExecStart=/usr/bin/nhid (code=exited, status=127) Main PID: 1119 (code=exited, status=127) CPU: 1ms
Sep 19 17:51:09 kam systemd[1]: nhid.service: Scheduled restart job, restart counter is at 5. Sep 19 17:51:09 kam systemd[1]: Stopped nhi daemon. Sep 19 17:51:09 kam systemd[1]: nhid.service: Start request repeated too quickly. Sep 19 17:51:09 kam systemd[1]: nhid.service: Failed with result 'exit-code'. Sep 19 17:51:09 kam systemd[1]: Failed to start nhi daemon.
root@kam:~# ls -la /tmp/nhi.log ls: cannot access '/tmp/nhi.log': No such file or directory
Have you installed some unusual stuff during debian 11 installation or after? Are you using amd64 CPU?
I would like to reproduce it in new virtual machine.
If you just simply execute /usr/bin/nhid
in bash terminal does it immediattly exit as well?
root@kam:~# /usr/bin/nhid /usr/bin/nhid: error while loading shared libraries: libbpf.so.1: cannot open shared object file: No such file or directory
Can you run apt-get install libbpf0
and then /usr/bin/nhid
root@kam:~# apt install libbpf0 Reading package lists... Done Building dependency tree... Done Reading state information... Done libbpf0 is already the newest version (1:0.3-2). libbpf0 set to manually installed. 0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. root@kam:~# /usr/bin/nhid /usr/bin/nhid: error while loading shared libraries: libbpf.so.1: cannot open shared object file: No such file or directory
Cna you run : find / -name libbpf.so.1
does it find anything? if so try adding /usr/lib
(or whereever libbpf.so.1 is installed) to LD_LIBRARY_PATH by export LD_LIBRARY_PATH="/usr/lib:$LD_LIBRARY_PATH" then run nhid
root@kam:~# find / -name libbpf.so.1 find: ‘/proc/sys/fs/binfmt_misc’: Permission denied find: ‘/sys/kernel/tracing’: Permission denied find: ‘/sys/kernel/debug’: Permission denied find: ‘/sys/kernel/config’: Permission denied find: ‘/dev/.lxc/sys/kernel’: Permission denied find: ‘/dev/.lxc/sys/power’: Permission denied find: ‘/dev/.lxc/sys/class’: Permission denied find: ‘/dev/.lxc/sys/devices’: Permission denied find: ‘/dev/.lxc/sys/dev’: Permission denied find: ‘/dev/.lxc/sys/hypervisor’: Permission denied find: ‘/dev/.lxc/sys/fs’: Permission denied find: ‘/dev/.lxc/sys/bus’: Permission denied find: ‘/dev/.lxc/sys/firmware’: Permission denied find: ‘/dev/.lxc/sys/block’: Permission denied find: ‘/dev/.lxc/sys/module’: Permission denied find: ‘/dev/.lxc/proc/fs’: Permission denied find: ‘/dev/.lxc/proc/bus’: Permission denied find: ‘/dev/.lxc/proc/irq’: Permission denied find: ‘/dev/.lxc/proc/spl’: Permission denied find: ‘/dev/.lxc/proc/sys’: Permission denied find: ‘/dev/.lxc/proc/tty’: Permission denied find: ‘/dev/.lxc/proc/acpi’: Permission denied find: ‘/dev/.lxc/proc/driver’: Permission denied find: ‘/dev/.lxc/proc/sysvipc’: Permission denied find: ‘/dev/.lxc/proc/pressure’: Permission denied find: ‘/dev/.lxc/proc/dynamic_debug’: Permission denied find: ‘/dev/.lxc/proc/1’: Permission denied find: ‘/dev/.lxc/proc/133’: Permission denied find: ‘/dev/.lxc/proc/147’: Permission denied find: ‘/dev/.lxc/proc/160’: Permission denied find: ‘/dev/.lxc/proc/162’: Permission denied find: ‘/dev/.lxc/proc/164’: Permission denied find: ‘/dev/.lxc/proc/165’: Permission denied find: ‘/dev/.lxc/proc/174’: Permission denied find: ‘/dev/.lxc/proc/176’: Permission denied find: ‘/dev/.lxc/proc/224’: Permission denied find: ‘/dev/.lxc/proc/475’: Permission denied find: ‘/dev/.lxc/proc/573’: Permission denied find: ‘/dev/.lxc/proc/969’: Permission denied find: ‘/dev/.lxc/proc/1171’: Permission denied
Try running find / -name libbpf.so.*
, does it fail to find anything as well?
root@kam:~# find / -name libbpf.so.1 find: ‘/proc/sys/fs/binfmt_misc’: Permission denied find: ‘/sys/kernel/tracing’: Permission denied find: ‘/sys/kernel/debug’: Permission denied find: ‘/sys/kernel/config’: Permission denied find: ‘/dev/.lxc/sys/kernel’: Permission denied find: ‘/dev/.lxc/sys/power’: Permission denied find: ‘/dev/.lxc/sys/class’: Permission denied find: ‘/dev/.lxc/sys/devices’: Permission denied find: ‘/dev/.lxc/sys/dev’: Permission denied find: ‘/dev/.lxc/sys/hypervisor’: Permission denied find: ‘/dev/.lxc/sys/fs’: Permission denied find: ‘/dev/.lxc/sys/bus’: Permission denied find: ‘/dev/.lxc/sys/firmware’: Permission denied find: ‘/dev/.lxc/sys/block’: Permission denied find: ‘/dev/.lxc/sys/module’: Permission denied find: ‘/dev/.lxc/proc/fs’: Permission denied find: ‘/dev/.lxc/proc/bus’: Permission denied find: ‘/dev/.lxc/proc/irq’: Permission denied find: ‘/dev/.lxc/proc/spl’: Permission denied find: ‘/dev/.lxc/proc/sys’: Permission denied find: ‘/dev/.lxc/proc/tty’: Permission denied find: ‘/dev/.lxc/proc/acpi’: Permission denied find: ‘/dev/.lxc/proc/driver’: Permission denied find: ‘/dev/.lxc/proc/sysvipc’: Permission denied find: ‘/dev/.lxc/proc/pressure’: Permission denied find: ‘/dev/.lxc/proc/dynamic_debug’: Permission denied find: ‘/dev/.lxc/proc/1’: Permission denied find: ‘/dev/.lxc/proc/133’: Permission denied find: ‘/dev/.lxc/proc/147’: Permission denied find: ‘/dev/.lxc/proc/160’: Permission denied find: ‘/dev/.lxc/proc/162’: Permission denied find: ‘/dev/.lxc/proc/164’: Permission denied find: ‘/dev/.lxc/proc/165’: Permission denied find: ‘/dev/.lxc/proc/174’: Permission denied find: ‘/dev/.lxc/proc/176’: Permission denied find: ‘/dev/.lxc/proc/224’: Permission denied find: ‘/dev/.lxc/proc/475’: Permission denied find: ‘/dev/.lxc/proc/573’: Permission denied find: ‘/dev/.lxc/proc/969’: Permission denied find: ‘/dev/.lxc/proc/1171’: Permission denied root@kam:~# find / -name libbpf.so.* /usr/lib/x86_64-linux-gnu/libbpf.so.0.3.0 /usr/lib/x86_64-linux-gnu/libbpf.so.0 find: ‘/proc/sys/fs/binfmt_misc’: Permission denied find: ‘/sys/kernel/tracing’: Permission denied find: ‘/sys/kernel/debug’: Permission denied find: ‘/sys/kernel/config’: Permission denied find: ‘/dev/.lxc/sys/kernel’: Permission denied find: ‘/dev/.lxc/sys/power’: Permission denied find: ‘/dev/.lxc/sys/class’: Permission denied find: ‘/dev/.lxc/sys/devices’: Permission denied find: ‘/dev/.lxc/sys/dev’: Permission denied find: ‘/dev/.lxc/sys/hypervisor’: Permission denied find: ‘/dev/.lxc/sys/fs’: Permission denied find: ‘/dev/.lxc/sys/bus’: Permission denied find: ‘/dev/.lxc/sys/firmware’: Permission denied find: ‘/dev/.lxc/sys/block’: Permission denied find: ‘/dev/.lxc/sys/module’: Permission denied find: ‘/dev/.lxc/proc/fs’: Permission denied find: ‘/dev/.lxc/proc/bus’: Permission denied find: ‘/dev/.lxc/proc/irq’: Permission denied find: ‘/dev/.lxc/proc/spl’: Permission denied find: ‘/dev/.lxc/proc/sys’: Permission denied find: ‘/dev/.lxc/proc/tty’: Permission denied find: ‘/dev/.lxc/proc/acpi’: Permission denied find: ‘/dev/.lxc/proc/driver’: Permission denied find: ‘/dev/.lxc/proc/sysvipc’: Permission denied find: ‘/dev/.lxc/proc/pressure’: Permission denied find: ‘/dev/.lxc/proc/dynamic_debug’: Permission denied find: ‘/dev/.lxc/proc/1’: Permission denied find: ‘/dev/.lxc/proc/133’: Permission denied find: ‘/dev/.lxc/proc/147’: Permission denied find: ‘/dev/.lxc/proc/160’: Permission denied find: ‘/dev/.lxc/proc/162’: Permission denied find: ‘/dev/.lxc/proc/164’: Permission denied find: ‘/dev/.lxc/proc/165’: Permission denied find: ‘/dev/.lxc/proc/174’: Permission denied find: ‘/dev/.lxc/proc/176’: Permission denied find: ‘/dev/.lxc/proc/224’: Permission denied find: ‘/dev/.lxc/proc/475’: Permission denied find: ‘/dev/.lxc/proc/573’: Permission denied find: ‘/dev/.lxc/proc/969’: Permission denied find: ‘/dev/.lxc/proc/1176’: Permission denied
try running ldconfig
no output root@kam:~# ldconfig root@kam:~#
try running nhid
root@kam:~# nhid nhid: error while loading shared libraries: libbpf.so.1: cannot open shared object file: No such file or directory
try adding /usr/lib/
to LD_LIBRARY_PATH
variable and export LD_LIBRARY_PATH
, then try running nhid
root@kam:~# export LD_LIBRARY_PATH="/usr/lib:$LD_LIBRARY_PATH" root@kam:~# nhid nhid: error while loading shared libraries: libbpf.so.1: cannot open shared object file: No such file or directory
Try running ldd /usr/bin/nhid
, whats the output?
root@kam:~# ldd /usr/bin/nhid linux-vdso.so.1 (0x00007ffcedfb6000) libbpf.so.1 => not found libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007ffa2cf88000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffa2cf6b000) libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007ffa2ce28000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa2cc54000) /lib64/ld-linux-x86-64.so.2 (0x00007ffa2cfaf000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa2cb10000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa2caec000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa2cae6000)
Interesting, so on my debian 11 ldd outputs:
linux-vdso.so.1 (0x00007ffc6bd44000)
libbpf.so.0 => /lib/x86_64-linux-gnu/libbpf.so.0 (0x00007fa398f5e000)
libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fa398f43000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa398f26000)
libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fa398de3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa398c0e000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa398fbe000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa398aca000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa398aa6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa398aa0000)
The argument of ldd is exactly the same binary as yours. As you can see in my case libbpf.so.0
is shown as dependency but in yours libbpf.so.1
I am not sure why does it happen, but I am going to research that (what's your version of ld, ldd and libbpf-dev apt package?)
For now I think that your best bet would be just compilling nhi from source by running make build-daemon
and replacing newly compiled nhi.bpf.o
in /etc/nhi
and nhid
in /usr/bin
, and finally starting systemctl nhid
it is lxc container, does it. make any difference?
whats your kernel version?
root@kam:~# uname -r 5.10.0-25-amd64
i assume you are running debian 11 as host, but what lxc image are you running?
same deb11 lxc launch images:debian/bullseye kam
For now I think that your best bet would be just compilling nhi from source by running make build-daemon and replacing newly compiled nhi.bpf.o in /etc/nhi and nhid in /usr/bin, and finally starting systemctl nhid
can you share steps for compiling from source?
You should install clang (simply by apt-get install clang) and it would be enough, and then just follow:
For now I think that your best bet would be just compilling nhi from source by running make build-daemon and replacing newly compiled nhi.bpf.o in /etc/nhi and nhid in /usr/bin, and finally starting systemctl nhid
EDIT:
also install apt-get install build-essential
installed clang , build-essential, restarted still not working, what are the steps to compile nhi from source
This is output from a. dedicated server root@stirshaken ~ # ldd /usr/bin/nhid linux-vdso.so.1 (0x00007ffd73855000) libbpf.so.1 => not found libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1c53c13000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1c53bf6000) libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f1c53ab3000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1c538df000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c53c3b000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1c5379b000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1c53777000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1c53771000)
to build nhid you need to clone this repository, cd to the repo and run make build-daemon, https://github.com/strang1ato/nhi/blob/main/Makefile#L5. Later:
make build-daemon and replacing newly compiled nhi.bpf.o in /etc/nhi and nhid in /usr/bin, and finally starting systemctl nhid
git clone https://github.com/strang1ato/nhi.git nhi2 cd nhi2 make build-daemon clang -Wall -g -O2 -target bpf -D__TARGET_ARCH_x86 -c daemon/src/nhi.bpf.c -o nhi.bpf.o clang -Wall -c daemon/src/nhi.c -o nhi.o clang -Wall -c daemon/src/utils.c -o utils.o clang -Wall -c daemon/src/sqlite.c -o sqlite.o clang -Wall nhi.o utils.o sqlite.o -lbpf -lelf -lz -lsqlite3 -o nhid
root@kam:~/nhi2# cp nhi.bpf.o /etc/nhi root@kam:~/nhi2# cp nhid /usr/bin/ root@kam:~/nhi2# systemctl restart nhid root@kam:~/nhi2# systemctl status nhid ● nhid.service - nhi daemon Loaded: loaded (/etc/systemd/system/nhid.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit-hit) since Tue 2023-09-19 20:44:34 UTC; 4s ago Process: 2009 ExecStart=/usr/bin/nhid (code=exited, status=0/SUCCESS) Main PID: 2009 (code=exited, status=0/SUCCESS) CPU: 409ms
Sep 19 20:44:34 kam systemd[1]: nhid.service: Scheduled restart job, restart counter is at 5. Sep 19 20:44:34 kam systemd[1]: Stopped nhi daemon. Sep 19 20:44:34 kam systemd[1]: nhid.service: Start request repeated too quickly. Sep 19 20:44:34 kam systemd[1]: nhid.service: Failed with result 'start-limit-hit'. Sep 19 20:44:34 kam systemd[1]: Failed to start nhi daemon.
I think I might need to reproduce this on my device, I will let you know how it will go.
So on debian 11 host machine, I created new container, by running: sudo lxc-create -t download -n somethingg -- -d debian -a amd64
and choosing bullseye version of the debian container.
After attaching, on the fresh container I ran following combination of commands:
4 sudo apt-get install binutils gawk sqlite3 libsqlite3-dev libbpf-dev
10 sudo apt-get install clang build-essential
13 sudo apt-get install git -y
15 git clone https://github.com/strang1ato/nhi.git
16 cd nhi/
18 make build-daemon
19 make build-cli
20 apt-get install go
21 apt-get install golang
22 make build-cli
25 sudo -E bash ./install
26 ldd /usr/bin/nhid
And the out put of ldd /usr/bin/nhid
is following:
linux-vdso.so.1 (0x00007ffdcdda8000)
libbpf.so.0 => /lib/x86_64-linux-gnu/libbpf.so.0 (0x00007f2f73a87000)
libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007f2f73a6c000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2f73a4f000)
libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f2f7390c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2f73738000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2f73acb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2f735f4000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2f735d0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2f735ca000)
So in my case ldd finds: libbpf.so.0 => /lib/x86_64-linux-gnu/libbpf.so.0 (0x00007f2f73a87000)
.
I can't tell why libbpf.so can't be found on your container.
when i run nhi fetch {-1} i get no such shell session: 0 source /etc/nhi/nhi.bash exists in ~/.bashrc , i am running everything as root what could be wrong? Also where is the history of commands stored? no log file in /tmp