netopeer2-server crash on try to listen from netopeer2-cli

[matkocc]

Hi, I am trying to uplift to new versions of libyang/sysrepo/netopeer2/libnetconf.

sysrepo 1.4.70 --> 1.4.104 netopeer2 1.1.39 --> 1.1.53 libyang 1.0.184 --> 1.0.215 libnetconf2 1.1.26 --> 1.1.36

For some reason, netopeer2-server crashed when I tried to bring session up with netopeer2-cli. I see that there is segmentation fault on the "get" event.

Here you can see logs:

on netopeer2-cli:

Last login: Thu Feb 11 10:28:06 2021
root@odusim:~# netopeer2-cli
> listen --login muser
Waiting 60s for an SSH Call Home connection on port 4334...
The authenticity of the host '::ffff:10.193.10.2' cannot be established.
ssh-rsa key fingerprint is 6d:3b:53:44:a7:27:af:bf:14:45:61:45:e0:3c:9f:29:0e:93:9c:30.
Are you sure you want to continue connecting (yes/no)? yes
muser@::ffff:10.193.10.2 password:
nc ERROR: Session 1: SSH channel poll error (Socket error: disconnected).
nc ERROR: Session 1: invalid session, discarding.
cmd_listen: Receiving SSH Call Home on port 4334 as user "muser" failed.
>

on server side:

[2021-02-11 10:30:28.470187] sim netopeer2-server[928]: Received an SSH message "request-service" of subtype "ssh-userauth".
[2021-02-11 10:30:28.471168] sim netopeer2-server[928]: Received an SSH message "request-auth" of subtype "none".
[2021-02-11 10:30:33.054435] sim netopeer2-server[928]: Received an SSH message "request-auth" of subtype "password".
[2021-02-11 10:30:33.065015] sim netopeer2-server[928]: User "muser" authenticated.
[2021-02-11 10:30:33.065811] sim netopeer2-server[928]: Received an SSH message "request-channel-open" of subtype "session".
[2021-02-11 10:30:33.066394] sim netopeer2-server[928]: Received an SSH message "request-channel" of subtype "subsystem".
[2021-02-11 10:30:33.071295] sim netopeer2-server[928]: Call Home client "192.168.10.1" session 1 established.
[2021-02-11 10:30:33.071403] sim netopeer2-server[928]: Session 10 (user "root", CID 7) created.
[2021-02-11 10:30:33.071507] sim netopeer2-server[928]: There are no subscribers for "ietf-netconf-notifications" notifications.
[2021-02-11 10:30:33.071541] sim netopeer2-server[928]: Generated new event (netconf-session-start).
[2021-02-11 10:30:33.203819] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.203826] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.203833] sim netopeer2-server[928]: Schema "ietf-netconf" was requested.
[2021-02-11 10:30:33.204747] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.204752] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.216232] sim netopeer2-server[928]: Session 1: thread 0 event new RPC.
[2021-02-11 10:30:33.218920] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.218926] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.218933] sim netopeer2-server[928]: Schema "ietf-netconf-acm" was requested.
[2021-02-11 10:30:33.219356] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.219369] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.223868] sim netopeer2-server[928]: Session 1: thread 2 event new RPC.
[2021-02-11 10:30:33.228005] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.228012] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.228021] sim netopeer2-server[928]: Schema "ietf-yang-library" was requested.
[2021-02-11 10:30:33.228258] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.228267] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.231065] sim netopeer2-server[928]: Session 1: thread 1 event new RPC.
[2021-02-11 10:30:33.232372] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.232377] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.232550] sim netopeer2-server[928]: Resolving unresolved data nodes and their constraints...
[2021-02-11 10:30:33.232555] sim netopeer2-server[928]: All data nodes and constraints resolved.
[2021-02-11 10:30:33.232632] sim netopeer2-server[928]: Published event "rpc" "/ietf-netconf:get" with ID 1 priority 0 for 1 subscribers.
[2021-02-11 10:30:33.232862] sim netopeer2-server[928]: Processing "/ietf-netconf:get" "rpc" event with ID 1 priority 0 (remaining 1 subscribers).
[2021-02-11 10:30:33.263321] sim netopeer2-server[928]: Successful processing of "rpc" event with ID 1 priority 0 (remaining 0 subscribers).
[2021-02-11 10:30:33.263781] sim kernel:[  118.892891] netopeer2-serve[930]: segfault at b69c7014 ip b7dc387f sp b71cc000 error 4 in libpthread-2.28.so[b7dbe000+f000]
[2021-02-11 10:30:33.263786] sim kernel:[  118.892996] Code: 39 ce 75 cd 8b 43 04 48 89 43 04 75 6d 31 ed 8d 73 14 65 89 35 78 00 00 00 65 a1 70 00 00 00 83 e0 fe 39 c6 0f 84 41 02 00 00 <8b> 08 83 e1 fe 39 ce 75 0a e9 de 02 00 00 8d 76 00 89 c1 8b 01 83

I also tried to upgrade from libssh v0.9.2 to libssh v0.9.5, but the same problem occurs.

Can you point where the problem is?

Thanks, Matko

[jktjkt]

Is your server on a different architecture than x86_64, perhaps? If so, then see sysrepo/sysrepo#2312, otherwsie, it's hard to tell without a proper backtrace; you'll need debug symbols (and possibly more) for that.

[michalvasko]

First, please update to the latest devel branch. Then, compile the server with debug symbols (the default) if you have not and try to run it with gdb (for the backtrace) or valgrind (for full crash output).

[rupali139]

Hi, can you please tell me the command to get netopeer2 server-side logs (I am using docker sysrepo image for netopeer2)

Regards Rupali