Interoperability error with Cisco Nexus NX-OS: NETCONF error: missing-attribute (rpc) - Wrong document: namespaces not specified

[jktjkt]

I'm using libnetconf from git (patched with https://github.com/CESNET/libnetconf/pull/203). When I try to talk to a Cisco Nexus 3172PQ (NX-OS 6.0(2)U5(4)), a simple get fails:

$ LD_LIBRARY_PATH=/opt/libnetconf/lib/ ~/work/prog/libnetconf/examples/apps/get -p 22 -l xifi taurida.vesnicky.cesnet.cz -d
Connecting to port 22 at taurida.vesnicky.cesnet.cz as user xifi.
libnetconf VERBOSE: Keyboard-interactive authentication
Password: 
libnetconf DEBUG: Writing message (session ): <?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <capabilities>
    <capability>urn:ietf:params:netconf:base:1.0</capability>
    <capability>urn:ietf:params:netconf:base:1.1</capability>
    <capability>urn:ietf:params:netconf:capability:writable-running:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:startup:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>
  </capabilities>
</hello>

libnetconf DEBUG: Received message (session ): <?xml version="1.0" encoding="ISO-8859-1"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <capabilities>
    <capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>
    <capability>urn:ietf:params:netconf:base:1.0</capability>
  </capabilities>
  <session-id>24939</session-id>
</hello>

libnetconf DEBUG: Writing message (session 24939): <?xml version="1.0" encoding="UTF-8"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
  <get/>
</rpc>

libnetconf DEBUG: Received message (session 24939): <?xml version="1.0" encoding="ISO-8859-1"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
  <rpc-error>
    <error-type>rpc</error-type>
    <error-tag>missing-attribute</error-tag>
    <error-severity>error</error-severity>
    <error-message>Wrong document: namespaces not specified</error-message>
    <error-info>
      <bad-element>rpc</bad-element>
    </error-info>
  </rpc-error>
</rpc-reply>

NETCONF error: missing-attribute (rpc) - Wrong document: namespaces not specified
libnetconf DEBUG: Writing message (session 24939): <?xml version="1.0" encoding="UTF-8"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
  <close-session/>
</rpc>

libnetconf DEBUG: Received message (session 24939): <?xml version="1.0" encoding="ISO-8859-1"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
  <ok/>
</rpc-reply>

Here's the OS details of that switch:

taurida# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
  BIOS:      version 2.7.0
  loader:    version N/A
  kickstart: version 6.0(2)U5(4)
  system:    version 6.0(2)U5(4)
  Power Sequencer Firmware: 
             Module 1: version v1.1
  BIOS compile time:       04/01/2014
  kickstart image file is: bootflash:///n3000-uk9-kickstart.6.0.2.U5.4.bin
  kickstart compile time:  8/3/2015 3:00:00 [08/03/2015 11:42:25]
  system image file is:    bootflash:///n3000-uk9.6.0.2.U5.4.bin
  system compile time:     8/3/2015 3:00:00 [08/03/2015 13:23:43]

Hardware
  cisco Nexus 3172 Chassis ("48x10GE + 6x40G Supervisor")
  Intel(R) Pentium(R) CPU  @ 2.00GHz
 with 3793756 kB of memory.
  Processor Board ID FOC18173KMS

  Device name: taurida
  bootflash:    1966080 kB

Kernel uptime is 286 day(s), 1 hour(s), 19 minute(s), 15 second(s)

Last reset at 703451 usecs after  Thu Sep  3 14:51:50 2015

  Reason: Reset Requested by CLI command reload
  System version: 6.0(2)U2(5)
  Service: 

plugin
  Core Plugin, Ethernet Plugin

I'm from CESNET, too, so I'll be happy to give you HW access if it's needed. Alternatively, I can pipe raw netconf through SSH for you if you have some debugging stuff.

[rkrejci]

I've found a similar issue on ODL, so you're not alone. Unfortunatelly, there is no answer/solution for that.

According to my (very old) experience with Cisco NETCONF implementation, they were used to have issues with processing namespaces. I've tried to check nexus's docs (just googled, you may have a better one) about its NETCONF interface, so I would try to follow it, if works at least what they write in their docs. Especially follow the way they use namespaces (the linked doc includes example with <get>). With the netopeer-cli, you are able to specify <rpc>'s content, so you can try to user-rpc command with the following data:

<nf:get xmlns:nf="urn:ietf:params:xml:ns:netconf:base:1.0"/>

Alternatively, you can try to communicate with the device directly via ssh:

$ ssh DEVICE_HOSTNAME -p 830 -s netconf
[here you will get <hello> from the device, type the following lines into the terminal]
<?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <capabilities>
    <capability>urn:ietf:params:netconf:base:1.0</capability>
  </capabilities>
</hello>]]>]]>
<?xml version="1.0" encoding="UTF-8"?>
<nf:rpc xmlns:nf="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
  <nf:get/>
</nf:rpc>]]>]]>

[here you should get data or error info from the device]

[jktjkt]

Thanks for the XML example, it seems that I was wrong in my analysis and that this bug has nothing to do with XML elements, after all. Using the example interaction provided by you doesn't change the NX-OS' behavior, the error message remains the same.

This example from a random blog worked:

<?xml version="1.0"?>
 <nc:rpc message-id="1" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0"
   xmlns="http://www.cisco.com/nxos:1.0:nfcli">
  <nc:get>
    <nc:filter type="subtree">
      <show>
          <vlan>
          </vlan>
      </show>
    </nc:filter>
  </nc:get>
 </nc:rpc>]]>]]>

Omitting the <vlan></vlan> resulted in another error message:

<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns="http://www.cisco.com/nxos:1.0:nfcli" message-id="1">
  <nc:rpc-error>
    <nc:error-type>application</nc:error-type>
    <nc:error-tag>invalid-value</nc:error-tag>
    <nc:error-severity>error</nc:error-severity>
    <nc:error-message>Syntax error while parsing 'show'

</nc:error-message>

...and using just an empty <nc:filter type="subtree"/> let into a terminating connection. Seems that their Netconf implementation is seriously broken.

Here's something useful:

<?xml version="1.0"?>
 <nc:rpc message-id="1" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0"
   xmlns="http://www.cisco.com/nxos:1.0:nfcli">
  <nc:get>
  </nc:get>
 </nc:rpc>]]>]]>

<?xml version="1.0" encoding="ISO-8859-1"?>
<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns="http://www.cisco.com/nxos:1.0:nfcli" message-id="1">
  <nc:rpc-error>
    <nc:error-type>application</nc:error-type>
    <nc:error-tag>missing-attribute</nc:error-tag>
    <nc:error-severity>error</nc:error-severity>
    <nc:error-message>rpc-get requires filter element</nc:error-message>
    <nc:error-info>
      <nc:err-element>get</nc:err-element>
    </nc:error-info>
  </nc:rpc-error>
</nc:rpc-reply>
]]>]]>

Swapping the actual XML namespaces doesn't change anything -- IOW, this generates the same result:

<?xml version="1.0"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:c="http://www.cisco.com/nxos:1.0:nfcli" message-id="1">
<get/>
</rpc>]]>]]>

So it seems that a mere presence of their proprietary namespace which is however not present in the initial <hello> is enough to change their behavior.

I do not desperately need to talk Netconf to this particular switch, I'm just getting to know the protocol and this HW was handy for experiments. Feel free to close this bugreport as a somebody else's problem. I'll look into upgrading the switch's FW and following with Cisco.