Get neighbour information is giving back an error message

[scrollins]

Hello again, I'm trying to pull neighbor information from netlink and I get an error message back. How do I debug what this error message means? This is the error that I'm seeing

root@ubuntu-bionic:/opt/netlink/target/debug/examples# ./dump_links
>>> NetlinkMessage { header: NetlinkHeader { length: 28, message_type: 30, flags: NetlinkFlags(1), sequence_number: 1, port_number: 0 }, payload: InnerMessage(GetNeighbour(NeighbourMessage { header: NeighbourHeader { family: 0, ifindex: 0, state: Incomplete, flags: NeighbourFlags(0), ntype: 0 }, nlas: [] })) }
<<< NetlinkMessage { header: NetlinkHeader { length: 48, message_type: 2, flags: NetlinkFlags(0), sequence_number: 1, port_number: 5364 }, payload: Error(ErrorMessage { code: -95, header: [28, 0, 0, 0, 30, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0] }) }

What I've basically done is modify the dump_links example to call GetNeighbor and pull the neighbor information. However, to construct a NeighborMessage, I have to construct the object and pass in the neighbor state which is undesirable. I want to pull back any neighbor where the neighbor flag = ROUTER and neighbor state can be anything i.e. REACHABLE, STALE, etc.

fn main() {
    let mut socket = Socket::new(Protocol::Route).unwrap();
    let _port_number = socket.bind_auto().unwrap().port_number();
    socket.connect(&SocketAddr::new(0, 0)).unwrap();

    let header = NetlinkHeader::new();
    let neighbor_message = NeighbourMessage {
        header: NeighbourHeader {
            family: 0,
            ifindex: 0,
            state: NeighbourState::Reachable,
            flags: NeighbourFlags::new(),
            ntype: 0
        },
        nlas: vec![]
    };

    let rtnl_message = RtnlMessage::GetNeighbour(neighbor_message);
    let mut packet = NetlinkMessage::new(header, NetlinkPayload::from(rtnl_message));

    packet.header.flags = NetlinkFlags::from(NLM_F_REQUEST);
    packet.header.sequence_number = 1;
    packet.finalize();
    let mut buf = vec![0; packet.header.length as usize];

    // Before calling serialize, it is important to check that the buffer in which we're emitting is big
    // enough for the packet, other `serialize()` panics.
    assert!(buf.len() == packet.buffer_len());
    packet.serialize(&mut buf[..]);

    println!(">>> {:?}", packet);
    socket.send(&buf[..], 0).unwrap();

    let mut receive_buffer = vec![0; 4096];
    let mut offset = 0;

    // we set the NLM_F_DUMP flag so we expect a multipart rx_packet in response.
    loop {
        let size = socket.recv(&mut receive_buffer[..], 0).unwrap();

        loop {
            let bytes = &receive_buffer[offset..];
            // Note that we're parsing a NetlinkBuffer<&&[u8]>, NOT a NetlinkBuffer<&[u8]> here.
            // This is important because Parseable<NetlinkMessage> is only implemented for
            // NetlinkBuffer<&'buffer T>, where T implements AsRef<[u8] + 'buffer. This is not
            // particularly user friendly, but this is a low level library anyway.
            //
            // Note also that the same could be written more explicitely with:
            //
            // let rx_packet =
            //     <NetlinkBuffer<_> as Parseable<NetlinkMessage>>::parse(NetlinkBuffer::new(&bytes))
            //         .unwrap();
            //
            let rx_packet: NetlinkMessage<RtnlMessage> =
                NetlinkMessage::deserialize(bytes).unwrap();

            println!("<<< {:?}", rx_packet);

            if rx_packet.payload == NetlinkPayload::Done {
                println!("Done!");
                return;
            }

            offset += rx_packet.header.length as usize;
            if offset == size || rx_packet.header.length == 0 {
                offset = 0;
                break;
            }
        }
    }
}

Sorry for pasting the entire thing but can you tell me if I'm going about this the right way and possibly what that error message means. I'm looking to do this by not using async/await which seems possible. Thanks

[little-dude]

Hello @scrollins,

I haven't tried the code yet but here are a couple general remarks to begin with:

• Filtering in netlink is buggy, and the general recommendation is to dump entire tables. This is what iproute2 does. For reference:
  • https://lists.infradead.org/pipermail/libnl/2013-June/001010.html
  • https://patchwork.ozlabs.org/patch/133440/
• You're using the NLM_F_REQUEST flag. You probably also need either NLM_F_MATCH or NLM_F_DUMP, although as per my point above, I would recommend going with NLM_F_DUMP since NLM_F_MATCH doesn't do what we think it does.
• Implementing a proper NetlinkError wrapper that implements Display and Debug is on my todo list. In the meantime, you can figure out the meaning of the errors code manually. The error code is a regular linux error code that you can look up. For instance, 95 is EOPNOTSUPP. Moreover, some errors have a specific meaning in the context of netlink. You can see how they map in the libnl source code: https://www.infradead.org/~tgr/libnl/doc/api/error_8c_source.html. Here, EOPNOTSUPP maps to NLE_OPNOTSUPP which is just Operation not supported.
• There are several ways to debug netlink issues, which are well explained in this blog post, and that I use myself to develop these libraries: https://jvns.ca/blog/2017/09/03/debugging-netlink-requests/. I usually create an nlmon interface, and capture the requests made by iproute2 or pyroute2 with tcpdump. Then I analyze the requests in wireshark. This is pretty tedious.

Finally, if you figure this out, do you mind contributing an example with Neighbor messages please? That would help tremendously because at the moment, I've mostly played with links and addresses.

[scrollins]

Thanks for the recommendations. I'll work through this and once(hopefully) I get it working, I'll gladly contribute back the example to your repo. Thanks again!

[little-dude]

Awesome :) Oh and if you're stuck I'll give it a try tonight anyway.

[scrollins]

👍 I plan on trying to get this to work over the weekend

[scrollins]

I've done a little debugging over the weekend. I changed the Netlink flag to use dump packet.header.flags = NetlinkFlags::from(NLM_F_DUMP);

I believe the issue are filters being sent in the message

NetlinkMessage { header: NetlinkHeader { length: 28, message_type: 30, flags: NetlinkFlags(768), sequence_number: 1, port_number: 0 }, payload: InnerMessage(GetNeighbour(NeighbourMessage { header: NeighbourHeader { family: 2, ifindex: 0, state: Reachable, flags: NeighbourFlags(0), ntype: 0 }, nlas: [] })) }

I believe the NeighborHeader shouldnt pass any filter values, get a full dump and match what i see when running ip neigh and then filtering can be done from there. Does that sound accurate to you?

[little-dude]

Does that sound accurate to you?

Yes it does :+1:

[little-dude]

@scrollins I've added a working example. You'll need to pull the latest master. See: https://github.com/little-dude/netlink/commit/1e5ea32503a8bfafbf622d132fbe0cb7a19381c4

[scrollins]

Thank you so much for pushing that change up. It was greatly appreciated. I was able to parse the message. One thing I noticed is that you have to set the offset if you want to get the payload of the message out of the byte array.

        loop {
            let bytes = &receive_buffer[offset..];
            let rx_packet: NetlinkMessage<RtnlMessage> =
                NetlinkMessage::deserialize(bytes).unwrap();
            //offset is 16 b/c netlink header is 128 bytes .  128/8 = 16
            let neighbor_bytes = &receive_buffer[offset + 16..];

            let neighbor_message = NeighbourMessageBuffer::new(neighbor_bytes);
...

I'm going to work on parsing it out and hopefully push up a PR that makes getting to the actual payload easier than having to put the offset in the loop. Thanks again for the changes. Cheers!

[little-dude]

When you call NetlinkMessage::deserialize(bytes).unwrap() you get a NetlinkMessage<RtnlMessage>, which has a payload attribute. The payload is already parsed, so you don't need to parse it yourself. I updated the example to show how to extract information from the payload in https://github.com/little-dude/netlink/commit/ec569ff752737320326f4ca943a18a768d3a0d85.

The new example now print the ARP table for IPv4 addresses:

$ ./dump_neighbours                                                                                                                                                                                                                                                                 
>>> NetlinkMessage { header: NetlinkHeader { length: 28, message_type: 30, flags: 769, sequence_number: 0, port_number: 0 }, payload: InnerMessage(GetNeighbour(NeighbourMessage { header: NeighbourHeader { family: 0, ifindex: 0, state: 0, flags: 0, ntype: 0 }, nlas: [] })) }                                            
IPv4 entries                                                                                                                                                                                                                                                                                                                  
10.0.5.153           e4:e7:49:71:4b:f8 (STALE)                                                                                                                                                                                                                                                                                
224.0.0.22           01:00:5e:00:00:16 (NOARP)                                                                                                                                                                                                                                                                                
10.0.5.101           00:20:6b:b8:18:8a (STALE)                                                                                                                                                                                                                                                                                
172.17.255.255       ff:ff:ff:ff:ff:ff (NOARP)                                                                                                                                                                                                                                                                                
255.255.255.255      ff:ff:ff:ff:ff:ff (NOARP)                                                                                                                                                                                                                                                                                
224.0.0.22           01:00:5e:00:00:16 (NOARP)                                                                                                                                                                                                                                                                                
0.0.0.0              00:00:00:00:00:00 (NOARP)                                                                                                                                                                                                                                                                                
239.255.255.250      01:00:5e:7f:ff:fa (NOARP)                                                                                                                                                                                                                                                                                
239.255.255.250      01:00:5e:7f:ff:fa (NOARP)                                                                                                                                                                                                                                                                                
10.0.7.255           ff:ff:ff:ff:ff:ff (NOARP)                                                                                                                                                                                                                                                                                
172.20.255.255       ff:ff:ff:ff:ff:ff (NOARP)                                                                                                                                                                                                                                                                                
172.18.255.255       ff:ff:ff:ff:ff:ff (NOARP)                                                                                                                                                                                                                                                                                
239.255.255.250      01:00:5e:7f:ff:fa (NOARP)                                                                                                                                                                                                                                                                                
224.0.0.22           01:00:5e:00:00:16 (NOARP)                                                                                                                                                                                                                                                                                
224.0.0.22           01:00:5e:00:00:16 (NOARP)                                                                                                                                                                                                                                                                                
239.255.255.250      01:00:5e:7f:ff:fa (NOARP)                                                                                                                                                                                                                                                                                
10.0.4.1             6c:3b:6b:f0:02:94 (REACHABLE)

Basically you can match against the payload:

let msg: NetlinkMessage<RtnlMessage> = NetlinkMessage::deserialize(bytes).unwrap();
match msg.payload {
    NetlinkPayload::Done => return,
    NetlinkPayload::InnerMessage(RtnlMessage::NewNeighbour(entry)) => {
        if entry.header.family as u16 == AF_INET {
            // do stuff with this NeighbourMessage
        }
    }
    NetlinkPayload::Error(err) => {
        eprintln!("Received a netlink error message: {:?}", err);
        return;
    }
    _ => {}
}

I'm sorry the API is so poorly documented :( Any help would be highly appreciated in that area.

[scrollins]

Awesome. I saw the payload was in there but I couldn't figure out how to parse it without setting an index on the byte array. I'd be more than glad to help document. I think the netlink linux man pages are really terrible and like your README states, you have to read multiple sources just to understand what the netlink protocol even does

[little-dude]

I saw the payload was in there but I couldn't figure out how to parse it without setting an index on the byte array

When I'll publish the new version of the crates you'll be able to see that on doc.rs, but for now you could just do cargo doc --open in the netlink-packet-route directory.

I think the netlink linux man pages are really terrible

It's not great yeah but we should also do a better job at documenting things on our side as well

[scrollins]

definitely agree. i can help out w/ the documenting once i get the netlink stuff working in my app. what are your thoughts on publishing the updated crate to crates.io? if not, i can use a git reference in my cargo.toml. also, i've been looking into a couple of things that i plan on implementing. all of the data structures defined in NLA enum are vec<8>s. for the destination, that will always be an ipv4 or ipv6 address. instead of having a vec<8>, you could define destination as https://doc.rust-lang.org/std/net/enum.IpAddr.html. the same for linklocaladdress, it could return a . macaddr struct instead of the vec. the hard part here is that the From trait on IpAddr only takes in an array. Its not hard to convert an array to a vec but that does mean copying data which I dont want to do

[little-dude]

i can help out w/ the documenting once i get the netlink stuff working in my app

That would be awesome, thank you!

what are your thoughts on publishing the updated crate to crates.io?

I'm just waiting for a new tokio release that uses futures 0.3. I think it will happen soon.

all of the data structures defined in NLA enum are vec<8>s. for the destination, that will always be an ipv4 or ipv6 address. instead of having a vec<8>, you could define destination as https://doc.rust-lang.org/std/net/enum.IpAddr.html. the same for linklocaladdress, it could return a . macaddr struct instead of the vec.

The reason for using Vec<u8> here is that I'm not 100% sure that these addresses will always be IP and MAC addresses. They are definitely the most common types of values, but Linux also supports other protocols with potentially different address structures (MPLS, GRE, etc.). I'm not opposed to parse these fields into proper IP and MAC addresses but I'd like to first make some research to see what other types of addresses we might encounter.

[scrollins]

According to the netlink documentation, dst should always be a ipv4 and ipv6 address. same w/ the link layer address http://man7.org/linux/man-pages/man7/rtnetlink.7.html NDA_DST a neighbor cache n/w layer destination address NDA_LLADDR a neighbor cache link layer address but that's only on neighbors. NLA looks like its used outside of neighbor though