Need to write to channel twice for packet to send

[Fumseck]

Hello,

I am facing strange behavior with this library while trying to implement a netconf client. I am on win 10.

the code is :

use ssh2::Session;
use std::error::Error;
use std::io::Read;
use std::io::Write;
use std::net::TcpStream;

fn main() -> Result<(), Box<dyn Error>> {
    let connect_string = format!("{}{}", "ios-xe-mgmt-latest.cisco.com", ":10000");

    let tcp = TcpStream::connect(connect_string)?;

    let mut sess = Session::new()?;

    let mut buf = String::new();

    sess.set_tcp_stream(tcp);

    sess.handshake()?;

    sess.userauth_password("developer", "C1sco12345")?;

    let mut channel = sess.channel_session()?;

    channel.subsystem("netconf");

    channel
        .write(
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
    <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
    <capabilities>
    <capability>
    urn:ietf:params:netconf:base:1.1
    </capability>
    </capabilities>
    </hello>
    ]]>]]>"
                .as_bytes(),
        )
        .expect("failed to write to channel");

    channel
        .write(
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
         <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
         <capabilities>
         <capability>
         urn:ietf:params:netconf:base:1.1
         </capability>
         </capabilities>
         </hello>
         ]]>]]>"
                .as_bytes(),
        )
        .expect("failed to write to channel ");

    //channel.flush();
    println!("debug marker");
    channel.read_to_string(&mut buf)?;
    println!("recv : {:?}", buf);

    Ok(())
}

A little bit about this code : it accesses a public always-on sandbox from cisco, so you can run it as-is, and the credentials are accessible to anyone who logs into their free devnet platform so i don't think writing them in cleartext here is an issue. Also, tcp/830 is the standard netconf port, but tcp/10000 is the correct one in this sandbox.

So basically i simply invoke the netconf subsystem, and send the hello message through the channel, expecting the server's hello in response.

The issue is that i need to write the hello message twice otherwise the message never sends (try removing one of the channel.write calls).

I have encountered similar issues when writing to files using a BufWrite, where the buffer would never be written unless you go over its size or call flush() explicitly, which is fine.

However, in the code above, adding a channel.flush() between the write and the debug marker does not seem to change the behavior.

I am a rust beginner so i could be wrong, but this does not seem right. Is someone able to explain this behavior ?

[wez]

Neither this crate nor the underlying library should need to write twice in that way, so this feels like it might be a protocol implementation issue.

I don't know anything much about netconf... I would suggest putting a newline on the end of the first write and then channel.flush() (removing the second write) and see if that makes things wake up on the server side.

[Fumseck]

Hello,

Thank you for taking the time to answer.

Still no luck with the following code :

use ssh2::Session;
use std::error::Error;
use std::io::Read;
use std::io::Write;
use std::net::TcpStream;

fn main() -> Result<(), Box<dyn Error>> {
    let connect_string = format!("{}{}", "ios-xe-mgmt-latest.cisco.com", ":10000");

    let tcp = TcpStream::connect(connect_string)?;

    let mut sess = Session::new()?;

    let mut buf = String::new();

    sess.set_tcp_stream(tcp);

    sess.handshake()?;

    sess.userauth_password("developer", "C1sco12345")?;

    let mut channel = sess.channel_session()?;

    channel.subsystem("netconf");

    channel
        .write(
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
    <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
    <capabilities>
    <capability>
    urn:ietf:params:netconf:base:1.1
    </capability>
    </capabilities>
    </hello>
    ]]>]]>\n"
                .as_bytes(),
        )
        .expect("failed to write to channel");

    channel.flush();
    println!("debug marker");
    channel.read_to_string(&mut buf)?;
    println!("recv : {:?}", buf);

    Ok(())
}

I took the opportunity to double check the RFCs for NETCONF and NETCONF over SSH. There is no indication that a newline is required at the end of the hello message.

However i noticed the RFC states :

Capabilities are advertised in messages sent by each peer during session establishment. When the NETCONF session is opened, each peer (both client and server) MUST send a element containing a list of that peer's capabilities.

If my understanding is correct this means the server's behavior should not change based on what/whether i write to the channel before reading, and the read_to_string() should just 'work' regardless.

I was able to verify that the implementation on cisco's side is correct with equivalent JS code that just waits for the server's hello without sending its own, but doing the same thing in rust with the above code minus write() and flush() does not print the server hello.

Could it be a sync/async or blocking/non-blocking issue ?

[crisidev]

I am from my phone, but I can tell you I have a quite similar version of your code running netconf without issues and without writing twice on the channel. I am still on version 0.3.3 and I am writing the whole netconf message using write_all, but I also used single writes with success..

[wez]

The issue is the use of read_to_string; that method will keep reading until EOF but because the channel is still alive, it blocks forever.

If you insert channel.send_eof()? before you flush you'll receive a response to your hello request... but that's likely not useful if you want to follow that up with some actual netconf dialogue.

Rather than using read_to_string, you should implement a read loop that accumulates data and searches for the message based delimiter so that your code knows that it has received the hello response, and then switch over to using the framing protocol described in the RFC.

[crisidev]

I have a read loop indeed in my code..

[Fumseck]

Hello,

Thank you very much for following up.

I have done some additional testing using your ideas and it looks like i indeed stupidly assumed the read_to_string() method would be able to magically know not to expect the EOF in the case of a subsystem.

Thanks again!

[crisidev]

Hello @Fumseck, no matter my comment above about the read loop, I am facing very similar issues and I basically can't get past the capabilities on Cisco on port 830.

Would it be possible to share a snippet with your solution to help me understand what am I doing wrong? It would be much appreciated :pray: :pray:

[crisidev]

Ah, I actually made it work.. It is incredible how fragile the Netconf implementation is in Cisco :(

Here is a snippet if anyone is interested:

#[macro_use]
extern crate log;
extern crate env_logger;

use ssh2::{Channel, Session};
use std::error::Error;
use std::io::prelude::*;
use std::io::Read;
use std::io::Write;
use std::net::{Shutdown, TcpStream};

const HELLO: &str = "<hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">
  <capabilities>
    <capability>urn:ietf:params:netconf:base:1.1</capability>
  </capabilities>
</hello>
]]>]]>";

const PAYLOAD_XML: &str = "<rpc xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\" message-id=\"102\">
  <get>
    <filter>
      <System xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\">
        <showversion-items/>
      </System>
    </filter>
  </get>
</rpc>";

const PAYLOAD: &str = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
    <rpc xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.1\" message-id=\"2\">
    <cli xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\"><mode>EXEC</mode><cmdline>show version</cmdline></cli>
</rpc>";

const READ_BUFFER_SIZE: usize = 16384;

fn read(channel: &mut Channel) -> String {
    let mut result = String::new();
    loop {
        let mut buffer = [1u8; 1];
        let bytes_read = channel.read(&mut buffer[..]).unwrap();
        let s = String::from_utf8_lossy(&buffer[..bytes_read]);
        result.push_str(&s);
        debug!("Read value: {}", s);
        if result.ends_with("]]>]]>") {
            error!("GOT HELLO, NOW GET OFF");
            break;
        }
        if result.ends_with("##") {
            error!("GOT THE REST, NOW GET OFF");
            break;
        }
        if bytes_read == 0 {
            warn!("Buffer is empty, SSH channel read terminated");
            break;
        }
    }
    result
}

fn main() -> Result<(), Box<dyn Error>> {
    env_logger::init();
    let tcp = TcpStream::connect("10:830")?;
    let mut sess = Session::new()?;
    sess.set_tcp_stream(tcp);
    sess.handshake().unwrap();
    sess.userauth_password("user", "password")?;
    assert!(sess.authenticated());

    let mut channel = sess.channel_session()?;
    channel.subsystem("netconf")?;
    let result = read(&mut channel);
    warn!("Result from connection:\n{}", result);

    let payload = format!("{}\n#{}\n{}\n##\n", HELLO, PAYLOAD.len(), PAYLOAD);

    info!("payload:\n{}", payload);
    let a = channel.write(payload.as_bytes())?;
    info!("Written {} bytes payload", a);
    let result = read(&mut channel);
    warn!("Result from payload:\n{}", result);

    // CLOSE SESSION
    channel.send_eof()?;
    channel.wait_eof()?;
    channel.close()?;
    channel.wait_close()?;
    Ok(())
}