Xmpp websocket connection not working

[sco0ter]

Original report by Anonymous.

---

I am trying to establish Xmpp websocket connection to my IoT gateway (previously working as Xmpp Bosh connection, which failed after a FW upgrade of my gateway). Trailing the websocket connection as websocket now is introduced during login (as identified during browser login / web developer tools).

So, I believe I have imported the correct packages required for websocket connection, and I believe I have a working setup for the Xmpp session:

#!java
//
m_WebSocketConfiguration = WebSocketConnectionConfiguration.builder()...
m_XmppConfiguration = XmppSessionConfiguration.builder()...
m_XmppClient = XmppClient.create("domain", m_XmppConfiguration, m_WebSocketConfiguration);
//
// Connect
try {
      m_XmppClient.connect();
}

However, as part of WebSocketConnectionConfiguration --> ClientEndpointConfig / XmppWebSocketDecoder (all pointing to javax.websocket) I end up with the following error:

#!java

[ERROR] [rnal.handler.FreeAtHomeBridgeHandler] - javax.websocket.DecodeException: unexpected element (uri:"http://etherx.jabber.org/streams", local:"stream"). Expected elements are...

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Just to follow up, hopefully you´ll be able to help. Trailing the websocket solution as when logging in to IoT gateway using mac Safari / web devoloper tool, xmpp-websocket turns up as type during login (port 5280). Also, when just testing in browser, running ipaddress:5280/xmpp-websocket/, the browser responds "It works! Now point your WebSocket client to this URL to connect to Prosody."

So in my java setup, as listed above, I am chasing the WebSocketConfiguration & XmppClient route. So, in addition to the code above, I have the following connection code:

#!java

try {
            m_WebSocketConfiguration.createConnection(m_XmppClient);
        }
       ..
       ..
try {
            String Adr = "domain";
            Jid JID = Jid.of(Adr);
            try {
                m_XmppClient.connect(JID);
            }
           ..
           ..
try {
            m_XmppClient.login("user", "password");
        }

And as mentioned in the previous post, I end up with error:

[ERROR] [rnal.handler.FreeAtHomeBridgeHandler] - javax.websocket.DecodeException: unexpected element (uri:"http://etherx.jabber.org/streams", local:"stream"). Expected elements are...

When logging in via web browser, the following stream is established:

#!xml

Websocket connection established

<stream:stream to="domain" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
<?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' from='domain id='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' xml:lang='en' xmlns='jabber:client'>
<stream:features xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mechanism>SCRAM-SHA-1</mechanism><mechanism>DIGEST-MD5</mechanism></mechanisms></stream:features>
<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='SCRAM-SHA-1'>KEY</aut>
<challenge xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>SOME_KEY</challenge>
<response xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>ANOTHER_KEY</response>
<success xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>YET_ANOTHER_KEY</success>
<stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="domain" version="1.0">
<?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' from='busch-jaeger.de' id=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx xml:lang='en' xmlns='jabber:client'>
<iq xmlns="jabber:client" type="set" id="bind_1"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><resource>426fc3</resource></bind></iq>
..
..
..

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

It seems that your server doesn't speak the correct WebSocket protocol as defined in https://tools.ietf.org/html/rfc7395

Opening the stream with is not allowed.

See here for the correct protocol flow: https://tools.ietf.org/html/rfc7395#section-3.4

(Client sends and server response with )

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

And I just realized (have no clue if it makes any difference):

The web browser opening stream uses " ", while there is a mix between ‘ ‘ and " " in the subsequent stream.

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

And also, I am using Xmpp.tryconnect(), and I see several examples using (ex issue #130).

The IoT server is not open source, but appears as Prosody server through web browser.

It all worked under the previous FW version of the IoT gateway, then using Xmpp BoshConfiguration / XmppClient.connect(). But then the connection also appeared as type http-bind when logging in via browser.

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

BOSH is a different protocol, your server seems to have it implemented correctly then.

"" or '' don't matter, XML allows both.

is simply not allowed in WebSocket connections. It's only used in plain ("normal") TCP connection on port 5222. See https://tools.ietf.org/html/rfc7395#section-3.3 The correct initial header is ``` ```

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Well, managed to read some data from browser .har file from websocket login via web browser: It appears that the gateway server is following a different protocol flow (ietf draft, also from 2014): https://tools.ietf.org/html/draft-ietf-xmpp-websocket-00#section-3.1

And here websocket connection is established with a stream as listed above:

#!java
<stream:stream xmlns:stream="http://etherx.jabber.org/streams"
                      xmlns="jabber:client"
                      to="example.com"
                      version="1.0">

Any tips/ideas for how to customize the existing Rocks XMPP library to fit the server´s response?

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

Ah interesting. I fear, the library is not customizable enough, to easily use another protocol for XMPP over WebSockets.

You could of course write your own Draft00WebSocketConnectionConfiguration, Draft00XmppWebSocketDecoder, Draft00XmppWebSocketEncoder and Draft00WebSocketConnection. It's mostly copy and paste from the existing classes.

But honestly, I'd rather put my energy into fixing the server so that it conforms to the final RFC 7395 spec.

EDIT: Already starting with draft 01 the protocol already used .

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Appreciate your feedback. Changing the server would be challenging as the server is a ABB brand IoT smarthome gateway (Free@Home).

Strange thing though, fiddling around I managed to connect/open some communication to the IoT gateway (appearing online). So now I have WebSocketConnectionConfiguration, XmppSessionConfiguration and XmppClient = ("domain", XmppSessionConfiguration, WebSocketConnectionConfiguration).

By running WebSocketConnectionConfiguration.createConnection(XmppClient) I got recognition of the gateway as online. However, still not able to read/send actual "data", and I get error as shown below. Trying XmppClient.login() for resource binding, but haven´t succeeded yet.

#!java

java.lang.IllegalStateException: Cannot send stanzas before resource binding has completed.
    at rocks.xmpp.core.session.XmppSession.sendInternal(XmppSession.java:885) ~[?:?]
    at rocks.xmpp.core.session.XmppSession.trackAndSend(XmppSession.java:1000) ~[?:?]
    at rocks.xmpp.core.session.XmppSession.sendIQ(XmppSession.java:973) ~[?:?]
    at rocks.xmpp.core.session.XmppSession.sendAndAwait(XmppSession.java:822) ~[?:?]
    at rocks.xmpp.core.session.XmppSession.query(XmppSession.java:748) ~[?:?]
    at rocks.xmpp.core.session.XmppSession.query(XmppSession.java:733) ~[?:?]
    at rocks.xmpp.extensions.rpc.RpcManager.call(RpcManager.java:113) ~[?:?]
    at org.openhab.binding.freeathome.internal.handler.FreeAtHomeBridgeHandler.setDataPoint(FreeAtHomeBridgeHandler.java:177) ~[?:?]

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

WebSocketConnectionConfiguration.createConnection(XmppClient) is not intended to be used directly, but only by XmppClient#connect()

Calling XmppClient#connect() does connect to the WebSocket endpoint on the WebSocket layer (that's why you probably see it online) and after that does some XMPP handshake, e.g. negotiating .

If you call login() directly, the stream headers are not exchanged and it probably fails therefore.

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Again, appreciate your effort to help a guy in need.

Have created my own "class/method caller" flow diagram based on my setup to see where I can make potential changes (at least for the initial stream, assuming the rest of the stream could be fetched by the listeners/negiotiator). I now have the ```WebSocketConnectionConfiguration, XmppSessionConfiguration and XmppClient = ("domain", XmppSessionConfiguration, WebSocketConnectionConfiguration)

But I am now aiming for ```XmppClient.connect(Jid from)``` and subsequently ```XmppClient.login("user", "pwd")```

However, based on my flow diagram, I do not see any obvious places where to replace <open> to <stream> (ref ietf documentation). You mentioned that it would be needed to adjust both WebSocketConnectionConfiguration, WebSocketDecoder, WebSocketEncoder and WebSocketConnection. However, WebSocketConnection is the only class making reference to"open". 

Hope you will bare with me, as I am a 1 month old Java / Xmpp experimenter.

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

WebSocketConnection sends the Open element, yes. You need to replace it with StreamHeader or simple pass the SessionOpen from argument to the send method.

XmppWebSocketEncoder eventually takes this element and encodes it to XML. You could check, if it's a StreamHeader and if yes, pass the XMLStreamWriter to the writeTo method. The goal is to return the StreamHeader as XML string from the encode method.

XmppWebSocketDecoder does the opposite. Unfortunately decoding is a little bit harder, especially if there's only a partial XML element (stream header). I suggest you use XMLEventReader or XMLStreamReader. Take a look at XmppStreamReader, which reads the header from an InputStream.

After decoding, the elements are passed to the message handler in WebSocketConnection again (private onRead method, which you need to adjust).

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Sorry for bothering you with a problem which really isn't a XMPP library problem, rather an issue for my so-called websocket server.

I have followed your tips for the initial output stream, whereas I now have XmppSession.tryconnect --> New WebSocketClientConnection --> WebSocketConnection.open(SessionOpen (= StreamHeader)) --> Return send (SessionOpen)

Furthermore, I have adjusted the code for XmppWebSocketEncoder to check if StreamElement is instance of StreamHeader:

#!java

public final String encode(final StreamElement object) throws EncodeException {
        try (Writer writer = new StringWriter()) {
            XMLStreamWriter xmlStreamWriter = null;
            try {
                if (object instanceof StreamHeader) {
                    object.writeTo(xmlStreamWriter);
                    marshaller.get().marshal(object, xmlStreamWriter);
                    String xml = xmlStreamWriter.toString();
                    if (interceptor != null) {
                        interceptor.accept(xml, object);
                    }
                    return xml;
                }
                else {
                    xmlStreamWriter = XmppUtils.createXmppStreamWriter(xmlOutputFactory.createXMLStreamWriter(writer), object instanceof StreamFeatures || object instanceof StreamError);
                    marshaller.get().marshal(object, xmlStreamWriter);
                    xmlStreamWriter.flush();
                    String xml = writer.toString();
                    if (interceptor != null) {
                    interceptor.accept(xml, object);
                    }
                    return xml;
                }
            }
                finally {
                if (xmlStreamWriter != null) {
                    xmlStreamWriter.close();
                }
            }
        } catch (Exception e) {
            throw new EncodeException(object, e.getMessage(), e);
        }
    }

However, with this adjustment, I still end up with the same error message:

#!java
rocks.xmpp.core.XmppException: javax.websocket.DecodeException: unexpected element (uri:"http://etherx.jabber.org/streams", local:"stream"). Expected elements are <{urn:xmpp:sm:3}a>,<{urn:ietf:params:xml:ns:xmpp-sasl}abort>.....

I have no clue if the error message relates to the initial output stream sent to the server, or if the server replies with an actual opening stream which just is decoded as error by the code.

I see from StreamHeader.writeTo that the method sets up startelement with the "http://etherx.jabber.org/streams" part as the first input. Not sure if sequence of elements would matter?

Based on webbrowser login, the following opening stream is actually happening:

#!xml

<stream:stream to="domain" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
<?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' from='domain id='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' xml:lang='en' xmlns='jabber:client'>

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

You need to adjust the Decoder, too. The exception occurs, because the current decoder expects full XML elements, (e.g. with start and end tag), which then are put into the unmarshaller and are unmarshalled to an Java object. However, is only an opening tag and cannot be put into the unmarshaller, but needs manual parsing instead.

The order of attributes doesn't matter.

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Just to ensure I understand it correctly, WebSocketEncoder handles client-server stream and WebSocketDecoder handles server-client stream? Meaning, based on the exception I get, my setup manages the client-server starting stream but struggle to handle the starting stream from the server?

I have had a look at the XmppStreamReader and WebSocketDecoder, but without understanding where the exception occur. XmppStreamReader seem to be able to handle through isStartElement (which is the actual opening stream from the server), and I do not see why WebSocketDecoder will throw an exception unless the unmarshaller itself creates such an exception? Not sure how to overcome this, as I assume any stream need to go via unmarshaller?

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

Yes, you understood correctly. The exception occurs in the XmppWebSocketDecoder#decode() method. The unmarshaller doesn't except an half-open element (without end tag), throws JAXBException, which is converted to DecodeException. Unmarshaller can only unmarshal complete XML (i.e. with start and end tag).

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Maybe we should just close this issue before I irritate you along with myself for the problems (and the programming incompetence on my behalf) I struggle with.

Just tried to bypass the XmppWebSocketDecoder Unmarshaller and create and return a manual StreamElement i.e. StreamHeader if receiving string contains 'stream:stream'. Just tested with a similar setup as the receiving opening stream from the server from browser login before actually trying to extract and manually add the actual 'id' (to see if the exception would change).

I.e.

#!java
 public final StreamElement decode(final String s) throws DecodeException {
        try (StringReader reader = new StringReader(s)) {
            String Element = IOUtils.toString(reader);
            if (Element.contains("stream:stream")) {
                if (Element.contains("/stream:stream")) {
                    StreamHeader close = new StreamHeader.CLOSING_STREAM_TAG();
                    if (onRead != null) {
                        onRead.accept(s, close);
                    }
                    return close;
                }
                else {
                    StreamHeader open = "<manual stream based on web browser login>";
                    if (onRead != null) {
                        onRead.accept(s, open);
                    }
                    return open;
                }    

            }
        else {
            StreamElement streamElement = (StreamElement) unmarshaller.get().unmarshal(reader);
            if (onRead != null) {
                onRead.accept(s, streamElement);
            }
            return streamElement;
        } 
    }   catch (JAXBException e) {
            logger.warn("Decoder Failure");
            throw new DecodeException(s, e.getMessage(), e);
        }
    }

However, with this code, I still get the exact same DecodeException:

#!java
rocks.xmpp.core.XmppException: javax.websocket.DecodeException: unexpected element (uri:"http://etherx.jabber.org/streams", local:"stream"). Expected elements are <{urn:xmpp:sm:3}a>,<{urn:ietf:params:xml:ns:xmpp-sasl}abort>.....

So it does not seem as it is the unmarshaller causing the problem (unless the String.contains() part is not recognized from my code).

So, going further, would you think this is a far-fetched problem to solve? I have no clue what additional problems I could meet if I actually was to solve the Websocket connection. As mentioned, I had working XMPP over BOSH connection, but not sure if the existing stanza code etc also would work over Websocket connection?

Please advice if you would recommend to stop chasing an unreachable solution.

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Btw, what would be the expected string (regular websocket/XMPP protocol) going as argument into the StreamElement decode method look like?

Just to know how the if should be built.

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

I think, you should debug your code and analyse some stacktrace, I can't really help here from remote. I don't even know, what libary you are using (Element.contains()). But theoretically is looks like a valid approach.

The string for the regular WebSocket connection would be <open ..../>.

[sco0ter]

Original comment by Stian Kjøglum (Bitbucket: kjoglum, GitHub: kjoglum).

---

Really appreciate your patience and your willingness to help.

I finally managed to adjust XmppWebSocketDecoder to work outside Unmarshaller and DecodeException if receiving string contains stream:stream. Meaning I extract the ID from the string and creates a new Open for the Websocket Model which is acknowledged by the code, and subsequent stream is flowing as it should.

So, I actually manage to connect to server and login/bind, and luckily, my "old code" for XMPP over BOSH is still working as it used to as well, so I am a happy guy (and potentially also others as part of the OpenHab community).

Really appreciate your guidance.

[sco0ter]

Original comment by Christian Schudt (Bitbucket: sco0ter, GitHub: sco0ter).

---

Your workaround is valid, too. If you could convince your server developer to just update to the official WebSocket spec, that would be even better, so that other libraries can use that server, too, without annoying modifications.