Not connecting to WSS Server - ESP8266

[0xinsanity]

Hello,

I recently started using this library and found a breaking issue. When trying to run with a test API like echo.websocket.org, I have had no issues getting responses, however when trying to test with finnhub.com's API (which includes query parameters which is my guess for why it is not connecting), it will not connect.

My code is below:

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <WebSocketsClient.h>

#include <Hash.h>

ESP8266WiFiMulti WiFiMulti;
WebSocketsClient webSocket;

void webSocketEvent(WStype_t type, uint8_t * payload, size_t length) {

  switch(type) {
    case WStype_DISCONNECTED:
      Serial.println("[WSc] Disconnected!\n");
      Serial.print(String((char *)payload));
      break;
    case WStype_CONNECTED: {
      Serial.println("[WSc] Connected to url:\n");
      //Serial.print(payload);

      // send message to server when Connected
      webSocket.sendTXT("{\"type\":\"subscribe\",\"symbol\":\"BINANCE:BTCUSDT\"}");
    }
      break;
    case WStype_TEXT:
      Serial.println("[WSc] get text: \n");
      Serial.print(String((char *)payload));
      //Serial.print(payload);

      // send message to server
      // webSocket.sendTXT("message here");
      break;
    case WStype_BIN:
      //Serial.println("[WSc] get binary length: %u\n", length);
      hexdump(payload, length);

      // send data to server
      // webSocket.sendBIN(payload, length);
      break;
    case WStype_PING:
        // pong will be send automatically
        Serial.println("[WSc] get ping\n");
        break;
    case WStype_PONG:
        // answer to a ping we send
        Serial.println("[WSc] get pong\n");
        break;
    }

}

void setup() {
  // USE_SERIAL.begin(921600);
  Serial.begin(115200);

  //Serial.setDebugOutput(true);
  Serial.setDebugOutput(true);

  for(uint8_t t = 4; t > 0; t--) {
    Serial.println("[SETUP] BOOT WAIT...\n");
    Serial.flush();
    delay(1000);
  }

  WiFiMulti.addAP("WifiNetwork", "Password");

  //WiFi.disconnect();
  while(WiFiMulti.run() != WL_CONNECTED) {
    delay(100);
  }

  // server address, port and URL
  webSocket.begin("ws.finnhub.io", 443, "?token={token id}");

  // event handler
  webSocket.onEvent(webSocketEvent);

  // try ever 5000 again if connection has failed
  webSocket.setReconnectInterval(5000);

  // start heartbeat (optional)
  // ping server every 15000 ms
  // expect pong from server within 3000 ms
  // consider connection disconnected if pong is not received 2 times
  webSocket.enableHeartbeat(15000, 3000, 2);

}

void loop() {
  webSocket.loop();
}

And the response is:

connected with WifiNetwork, channel 11
dhcp client start...
ip:192.168.1.148,mask:255.255.255.0,gw:192.168.1.254
[SETUP] BOOT WAIT...

[SETUP] BOOT WAIT...

[SETUP] BOOT WAIT...

[WSc] Disconnected!

pm open,type:2 0
[WSc] Disconnected!

Please, any help on this issue would be greatly appreciated.

[Links2004]

please enable debug output in the IDE including SSL/TLS. the setting can be found in the arduino IDE its in the setting where you configure your board. https://arduino-esp8266.readthedocs.io/en/latest/Troubleshooting/debugging.html

it is possible that the page is using a crypto that is not supported by the ESP SSL lib.

for more advanced IDEs you need to set a compile define options. for example platform IO:

build_flags = -D DEBUG_ESP_PORT=Serial -D DEBUG_ESP_SSL

[0xinsanity]

So these are the logs with debug mode on:

[WS-Client] connect ws...
[WS-Client] connected to stream.binance.com:9443.
[WS-Client][sendHeader] sending header...
[WS-Client][sendHeader] handshake GET /ws/btcusdt HTTP/1.1
Host: stream.binance.com:9443
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: 8s4yRcYD5ewUvKKpHzP7XQ==
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client

[write] n: 256 t: 10700
[WS-Client][sendHeader] sending header... Done (29263us).
[WS-Client] connection lost.
[WS-Client] client disconnected.
[WSc] Disconnected!

I guess the problem is that the full URL includes the "/ws/btcusdt" on the initial connect, so you cannot have it connect originally and then perform the GET aftewards. Anyway to connect everything into one instead of having the extra GET at the end? I did change the URL for this example by the way, but same thing should apply to both. The problem is the port number, because it's a mandatory requirement and will automatically be put at the end. Or perhaps I'm reading this incorrectly? Any ideas @Links2004 ?

[Links2004]

Websockets are a extension to HTTP / HTTPS so you always have a GET /path_here HTTP/1.1 in there. RFC: https://tools.ietf.org/html/rfc6455#section-1.2

what do you mean with The problem is the port number? are you referencing

Host: stream.binance.com:9443

if yes having the port number there is mandatory for non default ports (for http its 80 or https its 443). for default ports its optional but the RFC for HTTP 1.1 states that its ok and the server needs to handel it correctly.

based on the log you attached the TCP connection to stream.binance.com:9443 where opened and the ESP has send the HTTP header to open a websocket connection but the server the server simply closed the TCP connection with out sending any responce. with out the server logs its impossible to see what problem the server side has. the

GET /ws/btcusdt HTTP/1.1
Host: stream.binance.com:9443
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: 8s4yRcYD5ewUvKKpHzP7XQ==
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client

looks normal and good to me, what you can try is to disable the sending of the Origin since some bad server Implementation dont like it.

webSocket.setExtraHeaders("");

[0xinsanity]

Tried setting extra headers, but no luck. I guess by the problem being the port number, I think that to solve the issue, I might need the original request to be:

webSocket.begin("stream.binance.com:9443/ws/btcusdt");

Currently, it seems that it makes the original request and then adds the GET afterwards, which might be causing the problem. Is there anyway to do something like what I put above?

[Links2004]

I am not sure what you try to do, or what you mean with makes the original request and then adds the GET afterwards can you describe it in more detail?

calling begin like this:

webSocket.begin("stream.binance.com" , 9443, "stream.binance.com:9443/ws/btcusdt");

will result in

GET stream.binance.com:9443/ws/btcusdt HTTP/1.1
Host: stream.binance.com:9443
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: XXXXXXXXXXXXXXXXXXXXXXX
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client

which is not valid HTTP 1.1 but when it works for your strange server fin :)

[0xinsanity]

No, that's not what I meant. What I'm trying to get at is that the initial connection (using this library) to the websocket server seems to be stream.binance.com:9443 and then after the initial connect, it does a GET request to /ws/btcusdt. The websocket server I am trying to access will only work if you attach to it with the full url stream.binance.com:9443/ws/btcusdt rather than doing the two step process. Does that make sense?

[Links2004]

sorry this makes no sense to me right now. the lib opens on TCP connection and then sends the HTTP header over this one TCP connection. there are no 2 connections.

it is simply

• open TCP (SYN/ACK) --> connected to stream.binance.com:9443
• send HTTP header (one TCP packet) --> sending header... GET ....

normally the server response over the same TCP connection with a HTTP header but your server simply kills the connection. https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview#http_flow

the first two parameter of webSocket.begin are for the TCP connection the 3 is what is send in the HTTP header as url to the server.

stream.binance.com:9443 is a DNS/domain and port and the /ws/btcusdt is the path on the server where the TCP connection where made to.

RFC: https://tools.ietf.org/html/rfc6455#section-1.2

https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview#requests

[0xinsanity]

Hmm, I see. I guess what I just don't understand then is that all of the examples I have tried to use that have had an HTTP header after opening the TCP hasn't worked, but the one example I used that didn't have an HTTP header did work. The examples headers are as follows:

• wss://stream.binance.com:9443/ws/btcusdt
• wss://ws.finnhub.io:443/?token=c0c17g748v6o9159uf8g
• wss://data.alpaca.markets:443/stream

Do you think you could try getting any of these to work? All of them are giving me the same errors so I'm wondering if they all have the same issue and whether it is because I am doing something basic that is wrong. Also, thank you so much for helping me! This project is vital and I am really stuck.

[Links2004]

wss://stream.binance.com:9443/ws/btcusdt is dead:

wss://ws.finnhub.io:443/?token=c0c17g748v6o9159uf8g is broken 400 Bad Request

[WS-Client] connected to ws.finnhub.io:443.
[WS-Client][sendHeader] sending header...
[WS-Client][sendHeader] handshake GET ?token=c0c17g748v6o9159uf8g HTTP/1.1
Host: ws.finnhub.io:443
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: 653zS/V79vRFGPy15n5o2g==
User-Agent: arduino-WebSocket-Client

[write] n: 237 t: 6614
[WS-Client][sendHeader] sending header... Done (192528us).
[WS-Client][handleHeader] RX: HTTP/1.1 400 Bad Request
[WS-Client][handleHeader] RX: Server: nginx/1.10.3
[WS-Client][handleHeader] RX: Date: Fri, 05 Feb 2021 21:00:52 GMT
[WS-Client][handleHeader] RX: Content-Type: text/html
[WS-Client][handleHeader] RX: Content-Length: 173
[WS-Client][handleHeader] RX: Connection: close
[WS-Client][handleHeader] Header read fin.
[WS-Client][handleHeader] Client settings:
[WS-Client][handleHeader]  - cURL: ?token=c0c17g748v6o9159uf8g
[WS-Client][handleHeader]  - cKey: 653zS/V79vRFGPy15n5o2g==
[WS-Client][handleHeader] Server header:
[WS-Client][handleHeader]  - cCode: 400
[WS-Client][handleHeader]  - cIsUpgrade: 0
[WS-Client][handleHeader]  - cIsWebsocket: 1
[WS-Client][handleHeader]  - cAccept: 
[WS-Client][handleHeader]  - cProtocol: 
[WS-Client][handleHeader]  - cExtensions: 
[WS-Client][handleHeader]  - cVersion: 0
[WS-Client][handleHeader]  - cSessionId: 
[WS-Client][handleHeader] no Websocket connection close.

wss://data.alpaca.markets:443/stream works:

[WS-Client] connected to data.alpaca.markets:443.
[WS-Client][sendHeader] sending header...
[WS-Client][sendHeader] handshake GET /stream HTTP/1.1
Host: data.alpaca.markets:443
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: PO1OFQG95D/TWrJyvPY9eA==
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client

[write] n: 256 t: 6746
[WS-Client][sendHeader] sending header... Done (164117us).
[WS-Client][handleHeader] RX: HTTP/1.1 101 Switching Protocols
[WS-Client][handleHeader] RX: Date: Fri, 05 Feb 2021 20:58:13 GMT
[WS-Client][handleHeader] RX: Connection: upgrade
[WS-Client][handleHeader] RX: Upgrade: websocket
[WS-Client][handleHeader] RX: Sec-WebSocket-Accept: qeEHBWeUDf41RkTaFd9rGliJmPA=
[WS-Client][handleHeader] RX: Strict-Transport-Security: max-age=15724800; includeSubDomains
[WS-Client][handleHeader] Header read fin.
[WS-Client][handleHeader] Client settings:
[WS-Client][handleHeader]  - cURL: /stream
[WS-Client][handleHeader]  - cKey: PO1OFQG95D/TWrJyvPY9eA==
[WS-Client][handleHeader] Server header:
[WS-Client][handleHeader]  - cCode: 101
[WS-Client][handleHeader]  - cIsUpgrade: 1
[WS-Client][handleHeader]  - cIsWebsocket: 1
[WS-Client][handleHeader]  - cAccept: qeEHBWeUDf41RkTaFd9rGliJmPA=
[WS-Client][handleHeader]  - cProtocol: arduino
[WS-Client][handleHeader]  - cExtensions: 
[WS-Client][handleHeader]  - cVersion: 0
[WS-Client][handleHeader]  - cSessionId: 
[WS-Client][handleHeader] Websocket connection init done.
[WS][0][headerDone] Header Handling Done.
[WSc] Connected to url: /stream
[WS][0][sendFrame] ------- send message frame -------
[WS][0][sendFrame] fin: 1 opCode: 1 mask: 1 length: 9 headerToPayload: 0
[WS][0][sendFrame] text: Connected
[WS][0][sendFrame] pack to one TCP package...
[write] n: 15 t: 7022
[WS][0][sendFrame] sending Frame Done (175793us).
[HBtimeout] pong TIMEOUT! lp=-8297 millis=7203 pi=7203 count=1
[WS-Client] sending HB ping
[WS][0][sendFrame] ------- send message frame -------
[WS][0][sendFrame] fin: 1 opCode: 9 mask: 1 length: 0 headerToPayload: 0
[write] n: 6 t: 7222
[WS][0][sendFrame] sending Frame Done (137528us).
[WS][0][handleWebsocketWaitFor] size: 2 cWsRXsize: 0
[readCb] n: 2 t: 7371
[WS][0][handleWebsocketWaitFor][readCb] size: 2 ok: 1
[WS][0][handleWebsocket] ------- read massage frame -------
[WS][0][handleWebsocket] fin: 1 rsv1: 0 rsv2: 0 rsv3 0  opCode: 10
[WS][0][handleWebsocket] mask: 0 payloadLen: 0
[WS][0][handleWebsocket] get pong ()

but all showing other results then what your log show, so there must something broken in you setup.

[0xinsanity]

Do you mind sharing your setup? @Links2004

[Links2004]

ESP <--> AP <--> Switch <--> Router <--> Modem <---> Internet

code is: https://github.com/Links2004/arduinoWebSockets/blob/master/examples/esp8266/WebSocketClientSSL/WebSocketClientSSL.ino

I only changed

   webSocket.beginSSL("192.168.0.123", 81);

to the urls

   webSocket.beginSSL("stream.binance.com", 9443, "/ws/btcusdt");
   webSocket.beginSSL("ws.finnhub.io", 443, "/?token=c0c17g748v6o9159uf8g");
   webSocket.beginSSL("data.alpaca.markets", 443, "/stream");

for ws.finnhub.io I played around with

   webSocket.beginSSL("ws.finnhub.io", 443, "/?token=c0c17g748v6o9159uf8g", NULL, "");
   webSocket.setExtraHeaders("");

but still 400.

[0xinsanity]

I didn't realize I was supposed to use beginSSL instead of begin! That made everything work! Do you know what the difference between the two are?

[Links2004]

beginSSL is for wss / https which is encryptrd (SSL/TLS) begin is for ws / http which is plain text which you need is based on what the server is setup to use and can easy be seen be the URL :)

[0xinsanity]

Ohhh I see, thank you!

[smerch88]

Hey, Im trying this code and get respond

{"error":{"code":2,"msg":"Invalid request: missing field method at line 1 column 47"}}[WSc] Disconnected!

So how should I modify the url to let it work?

[smerch88]

Its working: webSocket.sendTXT("{\"method\": \"SUBSCRIBE\",\"params\":[\"ltcusdt@trade\"],\"id\": 1}");