search

CESNET / libyang

YANG data modeling language library
BSD 3-Clause "New" or "Revised" License
367 stars 292 forks source link

Crash inside libyang with filtering #1598

Closed vpestret closed 3 years ago

vpestret commented 3 years ago

Hi, I have Netopeer2 application and when tried filtering I've got crash get-config --source running --filter-subtree=tests/filters/oven.xml

oven.xml file contents:

<oven/>

stack trace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fe294523921 in __GI_abort () at abort.c:79
#2  0x00007fe29456c967 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fe294699b0d "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fe2945739da in malloc_printerr (str=str@entry=0x7fe294697d08 "free(): invalid pointer") at malloc.c:5342
#4  0x00007fe29457af0c in _int_free (have_lock=0, p=0x7fe294c4fd50 <__PRETTY_FUNCTION__.6809+16>, av=0x7fe2948cec40 <main_arena>) at malloc.c:4167
#5  __GI___libc_free (mem=0x7fe294c4fd60) at malloc.c:3134
#6  0x00007fe294b5fcd4 in dict_insert (ctx=0x55dbcf4a9dd0, value=0x7fe294c4fd60 "", len=0, zerocopy=1 '\001', str_p=0x7fe274025e08) at /opt/dev/libyang/src/hash_table.c:234
#7  0x00007fe294b5ffb7 in lydict_insert_zc (ctx=0x55dbcf4a9dd0, value=0x7fe294c4fd60 "", str_p=0x7fe274025e08) at /opt/dev/libyang/src/hash_table.c:299
#8  0x00007fe294b75e93 in lyd_create_opaq (ctx=0x55dbcf4a9dd0, name=0x7fe274001f20 "oven", name_len=4, prefix=0x7fe274001550 "", pref_len=0, 
    module_key=0x7fe274017910 "urn:ietf:params:xml:ns:netconf:base:1.0", module_key_len=39, value=0x7fe294c4fd60 "", value_len=0, dynamic=0x7fe292c6c61b "\001\342\177", format=LY_VALUE_XML, 
    val_prefix_data=0x0, hints=0, node=0x7fe292c6c630) at /opt/dev/libyang/src/tree_data.c:844
#9  0x00007fe294b8f2f3 in lyb_parse_subtree_r (lybctx=0x7fe27401fdc0, parent=0x0, first_p=0x7fe292c6c928, parsed=0x7fe292c6c7c0) at /opt/dev/libyang/src/parser_lyb.c:793
#10 0x00007fe294b901db in lyd_parse_lyb (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6c928, in=0x7fe27400ad40, parse_opts=458752, val_opts=0, data_type=LYD_TYPE_DATA_YANG, 
    parsed=0x7fe292c6c7c0, lydctx_p=0x7fe292c6c7b0) at /opt/dev/libyang/src/parser_lyb.c:1100
#11 0x00007fe294b74449 in lyd_parse (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6c928, in=0x7fe27400ad40, format=LYD_LYB, parse_opts=458752, val_opts=0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:366
#12 0x00007fe294b74966 in lyd_parse_data (ctx=0x55dbcf4a9dd0, parent=0x0, in=0x7fe27400ad40, format=LYD_LYB, parse_options=458752, validate_options=0, tree=0x7fe292c6c928)
    at /opt/dev/libyang/src/tree_data.c:435
#13 0x00007fe294b749e5 in lyd_parse_data_mem (ctx=0x55dbcf4a9dd0, data=0x7fe27401fc10 "lyb\020", format=LYD_LYB, parse_options=458752, validate_options=0, tree=0x7fe292c6c928)
    at /opt/dev/libyang/src/tree_data.c:446
#14 0x00007fe294b8f792 in lyb_parse_subtree_r (lybctx=0x7fe274011f40, parent=0x7fe274025bb0, first_p=0x0, parsed=0x0) at /opt/dev/libyang/src/parser_lyb.c:862
#15 0x00007fe294b8f50a in lyb_parse_subtree_r (lybctx=0x7fe274011f40, parent=0x0, first_p=0x7fe292c6cbd0, parsed=0x7fe292c6cbe0) at /opt/dev/libyang/src/parser_lyb.c:824
#16 0x00007fe294b901db in lyd_parse_lyb (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6cbd0, in=0x7fe2740018f0, parse_opts=196608, val_opts=0, data_type=LYD_TYPE_RPC_YANG, 
    parsed=0x7fe292c6cbe0, lydctx_p=0x7fe292c6cbc8) at /opt/dev/libyang/src/parser_lyb.c:1100
#17 0x00007fe294b74ffe in lyd_parse_op_ (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, in=0x7fe2740018f0, format=LYD_LYB, data_type=LYD_TYPE_RPC_YANG, tree=0x7fe292c6ccb0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:572
#18 0x00007fe294b752b8 in lyd_parse_op (ctx=0x55dbcf4a9dd0, parent=0x0, in=0x7fe2740018f0, format=LYD_LYB, data_type=LYD_TYPE_RPC_YANG, tree=0x7fe292c6ccb0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:625
#19 0x00007fe2949232c0 in sr_shmsub_rpc_listen_process_rpc_events (rpc_subs=0x55dbcf4d8100, conn=0x55dbcf4a9bc0) at /opt/dev/sysrepo/src/shm_sub.c:3129
#20 0x00007fe2948e2b2f in sr_process_events (subscription=0x55dbcf6422e0, session=0x0, stop_time_in=0x7fe292c6cde0) at /opt/dev/sysrepo/src/sysrepo.c:3060
#21 0x00007fe29492447f in sr_shmsub_listen_thread (arg=0x55dbcf6422e0) at /opt/dev/sysrepo/src/shm_sub.c:3575
#22 0x00007fe294e906db in start_thread (arg=0x7fe292c6d700) at pthread_create.c:463
#23 0x00007fe29460471f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

my Dockerfile fragment:

# libssh
RUN \
      cd /opt/dev && \
      git clone https://git.libssh.org/projects/libssh.git && cd libssh && \
      git checkout stable-0.9 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE="Release" -DWITH_ZLIB=ON -DWITH_NACL=OFF -DWITH_PCAP=OFF .. && \
      make -j && \
      make install

# libyang
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libyang.git && cd libyang && \
      git checkout v2.0.7 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# sysrepo
RUN \
      cd /opt/dev && \
      git clone https://github.com/sysrepo/sysrepo.git && cd sysrepo && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# libnetconf2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libnetconf2.git && cd libnetconf2 && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake  -DCMAKE_BUILD_TYPE:String="Debug" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DENABLE_BUILD_TESTS=OFF .. && \
      make -j && \
      make install

# netopeer 2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/Netopeer2.git && cd Netopeer2 && \
      git checkout v2.0.0 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install
vpestret commented 3 years ago

Here is full Dockerfile attached

FROM ubuntu:18.04

# defaults
RUN \
    apt-get update && apt-get install -y \
    git \
    curl \
    wget \
    libssl-dev \
    libtool \
    build-essential \
    vim \
    autoconf \
    automake \
    pkg-config \
    libgtk-3-dev \
    make \
    valgrind \
    doxygen \
    libev-dev \
    libpcre2-dev \
    unzip \
    sudo \
    python3 \
    bison \
    flex \
    swig \
    libcmocka0 \
    libcmocka-dev \
    cmake \
    inetutils-syslogd \
    libzmq3-dev \
    supervisor

# Adding netconf user
RUN adduser --system netconf
RUN mkdir -p /home/netconf/.ssh
RUN echo "netconf:1" | chpasswd && adduser netconf sudo

# Clearing and setting authorized ssh keys
RUN \
    echo '' > /home/netconf/.ssh/authorized_keys && \
    ssh-keygen -A && \
    ssh-keygen -t rsa -b 4096 -P '' -f /home/netconf/.ssh/id_rsa && \
    cat /home/netconf/.ssh/id_rsa.pub >> /home/netconf/.ssh/authorized_keys

# Updating shell to bash
RUN sed -i s#/home/netconf:/bin/false#/home/netconf:/bin/bash# /etc/passwd

RUN mkdir /opt/dev && sudo chown -R netconf /opt/dev

# set password for user (same as the username)
RUN echo "root:1" | chpasswd

# libssh
RUN \
      cd /opt/dev && \
      git clone https://git.libssh.org/projects/libssh.git && cd libssh && \
      git checkout stable-0.9 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE="Release" -DWITH_ZLIB=ON -DWITH_NACL=OFF -DWITH_PCAP=OFF .. && \
      make -j && \
      make install

# libyang
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libyang.git && cd libyang && \
      git checkout v2.0.7 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# sysrepo
RUN \
      cd /opt/dev && \
      git clone https://github.com/sysrepo/sysrepo.git && cd sysrepo && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# libnetconf2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libnetconf2.git && cd libnetconf2 && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake  -DCMAKE_BUILD_TYPE:String="Debug" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DENABLE_BUILD_TESTS=OFF .. && \
      make -j && \
      make install

# netopeer 2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/Netopeer2.git && cd Netopeer2 && \
      git checkout v2.0.0 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

ENV EDITOR vim
EXPOSE 830

COPY tests/oven.yang /home/netconf
RUN sysrepoctl -i /home/netconf/oven.yang

COPY configs/supervisord-m-plane.conf /etc/supervisord.conf

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]

also supervisor file

[supervisord]
nodaemon=true
logfile=/var/log/supervisord.log
loglevel=debug

[program:syslogd]
command=/usr/sbin/syslogd
autorestart=false
startsecs=0
redirect_stderr=true
priority=3

[program:netopeer2-server]
command=/usr/bin/netopeer2-server -d
autorestart=true
redirect_stderr=true
priority=3

enjoy!

by the way is there a documentation about libyang library architecture?

michalvasko commented 3 years ago

Okay, the crash should be fixed but note that you should not use subtree filters with elements without a namespace. In all likelihood they inherit some namespace from the RPC, which is not what you want.

is there a documentation about libyang library architecture?

I am not sure libyang has some specific architecture. But the documentation is here.

vpestret commented 3 years ago

Thank you very much for looking into this. Thank you for the good advice about namespaces. I've found this documentation already and started to study, in fact, there was a reference from the main page. As much as I can I'm glad to help improving such a great project!

vpestret commented 3 years ago

The bugfix provided helped to fix the crash asked but there is another crash appear:

#0  0x000055e5ac61df27 in filter_xpath_buf_append_node (node=0x7f3e58025db0, buf=0x7f3e6b4b4a88, size=1) at /opt/dev/Netopeer2/src/common.c:730
#1  0x000055e5ac61e0d8 in filter_xpath_buf_add_r (node=0x7f3e58025db0, buf=0x7f3e6b4b4a88, size=1, filter=0x7f3e6b4b4b30) at /opt/dev/Netopeer2/src/common.c:761
#2  0x000055e5ac61e368 in op_filter_subtree2xpath (node=0x7f3e58025db0, filter=0x7f3e6b4b4b30) at /opt/dev/Netopeer2/src/common.c:837
#3  0x000055e5ac61f388 in np2srv_rpc_get_cb (session=0x7f3e58002370, UNUSED_sub_id=113, op_path=0x55e5acd51670 "/ietf-netconf:get-config", input=0x7f3e58025bb0, event=SR_EV_RPC, UNUSED_request_id=1, 
                                                                     output=0x7f3e5800ad40, UNUSED_private_data=0x0) at /opt/dev/Netopeer2/src/netconf.c:249
#4  0x00007f3e6d16a7d9 in sr_shmsub_rpc_listen_call_callback (rpc_sub=0x55e5acdcb670, ev_sess=0x7f3e58002370, input_op=0x7f3e58025bb0, event=SR_SUB_EV_RPC, request_id=1, output_op=0x7f3e6b4b4cc0, 
                                                                                      err_code=0x7f3e6b4b4c88) at /opt/dev/sysrepo/src/shm_sub.c:2897
#5  0x00007f3e6d16b528 in sr_shmsub_rpc_listen_process_rpc_events (rpc_subs=0x55e5acd4e100, conn=0x55e5acd1fbc0) at /opt/dev/sysrepo/src/shm_sub.c:3182
#6  0x00007f3e6d12ab2f in sr_process_events (subscription=0x55e5aceb82e0, session=0x0, stop_time_in=0x7f3e6b4b4de0) at /opt/dev/sysrepo/src/sysrepo.c:3060
#7  0x00007f3e6d16c47f in sr_shmsub_listen_thread (arg=0x55e5aceb82e0) at /opt/dev/sysrepo/src/shm_sub.c:3575
#8  0x00007f3e6d6d86db in start_thread (arg=0x7f3e6b4b5700) at pthread_create.c:463
#9  0x00007f3e6ce4c71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Anyway when I've tried to supply correct subtree filter file with namespaces both crashes not happen.

<oven xmlns="urn:sysrepo:oven"/>
michalvasko commented 3 years ago

You just need to update netopeer2, I have already fixed that, too, just not mentioned it.

vpestret commented 3 years ago

When I've checked out Netopeer2 version 91f862f94f60b9bc112f6810e8fe293b18a8d9a5 (VERSION bump to version 2.0.2) and libyang to version 0b8573eb2b9acdab119823c143b47d06af76b1b1 (VERSION bump to version 2.0.8 ) then it worked thanks!!!