Could not query exchange for pair

[lolero]

I am getting a lot of the following warnings:

|WARN| Could not query exchange for pair <pair key> (might be an invalid pair) : err = 'There was an error invoking Hub method 'c2.queryexchangestate'.'

They happen after the gateway has already been able to subscribe to the pairs so I already have valid data. The problem is the data that I have received and think is currently valid is outdated.

Do you have an idea of why this is happening or how to fix it?

Would it be possible to emit a notification on the websocket to let the subscribed party know that the gateway is not able to query the exchange for that pair until updated data starts flowing again?

[aloysius-pgast]

This is a known problem on Bittrex side. See https://github.com/aloysius-pgast/bittrex-signalr-client/issues/17 and https://github.com/Bittrex/bittrex.github.io/issues/50#issuecomment-480890809

[aloysius-pgast]

Which pair did you have a problem with ?

[lolero]

theres many that intermittently trigger the error. i understand that the issue happens but it would be great if the error could trigger a notification so that my client is aware that the trades it calculates on the pairs that are not receiving current data should be ignored.

[aloysius-pgast]

How many pairs are you monitoring at the same time ?

[lolero]

100-200. is that a bottleneck?

[aloysius-pgast]

I think so. 10-15 pairs sounds more realistic.

[lolero]

is the limitation that wouldnt allow me to monitor 200 pairs on bittrex side? would monitoring less pairs guarantee that i wont get that error?

If the chances of the error happening are less but still exist, that would mean that i still risk placing trades based on outdated data :/

[aloysius-pgast]

Market subscriptions generates a lot of data (because of order books). See https://github.com/aloysius-pgast/bittrex-signalr-client/issues/23. That's the bottleneck. What are you aiming at exactly ? (maybe we can find an alternative)

[lolero]

I am trying to write a neural network that generates short term trades based on high volume and fast price movement.

It makes sense that my trying to monitor too many pairs causes this error and I will try monitoring less. However, if it still happens while I monitor less pairs without the gateway notifying me when it happens, it would basically wipe out the small margin of my potential trades :/

So monitoring less pairs is a good temporary solution but I still wouldn't be able to run such a system live unattended if this error occurs for a pair and the system cannot prevent the affected pairs' trades from going out to the exchange.

How cumbersome would you think it is to emit a notification when a pair subscription is affected?

[aloysius-pgast]

It should be possible to add a new type of subscription for each exchange, to account for this kind of custom notifications

[lolero]

Wouldn't a general notification for pair subscription interruptions be applicable to all exchanges?

It seems like there could be many reasons why pair subscriptions could have issues in other exchanges as well, and consumers would profit from getting promptly notified.

[aloysius-pgast]

Problem is that pair subscription interruptions doesn't really make sense per se and cannot really be generalized to all exchanges. It requires context which is only (easily) known to client. Custom exchanges events such as Bittrex QueryExchangeState problem is easier to define on the other end

[lolero]

I agree that subscriptions to pair data feed interruptions don't make sense. But I don't think that a user should get subscribed to interruption notifications in order to get notified when such interruptions happen.

But I get your point that leaving it up to the user is perhaps a more general and flexible approach.

The thing is that I cannot think of a use case where a consumer that subscribes to orderBook or trade data would not want to get a notification when a connection issue, like the Bittrex QueryExchangeState error, happens. It seems to me that anyone who is gathering data either for analysis or live trading would want to know if the data that they have is real-time or not. So I would vote for making the interruption subscription a default for all other subscriptions. On the other hand, a flag could be added for those that don't want to get interruption notifications to opt out of it.

[aloysius-pgast]

What I meant was that data feed interruptions does not make sense since its very context dependant, and the context is known by the user. 5 minutes without book order data might be perfectly valid for a given pair on a given exchange, while on another exchange for the same pair more than 30s without data might indicate a pb.

[lolero]

Oh. I think I had misunderstood the QueryExchangeState error. I didn't know that the error was the result of a timeout. I thought it was caused by the gateway getting an actual query exchange error from Bittrex, which is why I figured that forwarding such errors as notifications to the client would be useful.

So what are the conditions for the Bittrex QueryExchangeState error warning to be printed on the gateway's console?

[aloysius-pgast]

Bittrex QueryExchangeState error can be an error indicating :

• that a pair is not supported by Bittrex anymore. This can be temporary during maintenance window (that's why it is better to always retrieve the list of supported pair using REST API before subscribing
• that something wrong occured on Bittrex side of the connection (see https://github.com/Bittrex/bittrex.github.io/issues/50#issuecomment-480890809)

It is also possible to have timeout where Bittrex just stop emitting data and no error is returned at all

[lolero]

Ok. But then it is a product of the exchange issuing some sort of error. In this case it is Bittrex saying that the pair is temporarily not supported or that their side is having issues with the connection. For other exchanges it could be another sort of problem.

What I was proposing is that when the gateway gets errors from the exchange regarding a specific pair, as it does from Bittrex in this particular case, it emits a notification to the client with the corresponding pair and error.

That way, if and when any exchange has errors with any pair's data feed, the client is made aware right away.

[aloysius-pgast]

Yes, that's what I meant with the custom notification type, which would be dependant on the exchange. By subscribing to this custom type, a client could receive a notification similar to :

{
    "n":"custom",
    "d":{
        "exchange":"bittrex",
        "timestamp":1509986841.91
        "type":"QueryExchangeState",
        "data":{
            "pair":"USDT-BTC"
        }
    }
}

[lolero]

Ok. I see what you meant. That is fine with me but I think perhaps a more meaningful name for the notification would make it easier to follow for consumers. What about 'error'?

'custom' seems vague.

[aloysius-pgast]

I meant custom because each event is a custom which might be completely distinct for each exchange but on second thought, it would probably better to use something like streamError which would cover QueryExchangeState & anything related to websocket endpoints being unreachable (ie: connection error)

[lolero]

that sounds perfect!

[lolero]

Actually, it would also be very useful to get notifications when the gateway has to disconnect and retry the connection. I often get a 1590225908654|INFO|Connection (bittrex|2) will be disconnected in the gateway's console but not notification through the socket. This means that the order books that I have cached, may no longer be up to date when the socket reconnects and I am getting trades that experience high slippage because the rates that triggered them are no longer available.

Would it be possible to add notifications for these gateway disconnects as well so that I can clear my order books when it happens?

I think when such reconnections happen, the order books that are subscribed get re-synced. Is that correct?

[aloysius-pgast]

Yes orderbooks will be re-synced and you'll get an orderBook event first. Bittrex connection is disconnected in following cases :

• no more subscription exists for any pair
• connection has been closed remotely (in such case connection will be reconnected if at least one subscriptions exist for a pair) The message 1590225908654|INFO|Connection (bittrex|2) will be disconnected seems to indicate this is case 1 and is expected behaviour

[lolero]

What does 'no more subscription exists for any pair' mean exactly.

I am getting the disconnection while I am subscribed to order books of several pairs.

And the problem I am having is that when re-connection happens and the pairs start getting re-synced, the new order books trigger trades that are calculated based on order books of pairs that are outdated because their re-syncing has not happened just yet.

So it would be very useful for my system to become aware of a disconnection so that I can clear all order book cache when the disconnection happens and let the order books get re-synced from scratch when the gateway connects again.

[aloysius-pgast]

When the socket is reconnected, you should get an orderBook event which means you can forget cached order book.

[lolero]

Yes, and I do flush the cache when an 'orderBook' notification is received. The thing is that in my system, one pair's activity can affect another pair's neural network :/

So when a disconnection happens and the first pair's 'orderBook' notification arrives, it could trigger a trade on another pair, whose order book is outdated because the 'orderBook' for that pair has not arrived yet after the reconnection.

So basically any system that uses other pairs activity as trading indicators, would be affected by not being aware that a disconnection happened.

[lolero]

Yes orderbooks will be re-synced and you'll get an orderBook event first. Bittrex connection is disconnected in following cases :

* no more subscription exists for any pair

* connection has been closed remotely (in such case connection will be reconnected if at least one subscriptions exist for a pair)
  The message `1590225908654|INFO|Connection (bittrex|2) will be disconnected` seems to indicate this is case 1 and is expected behaviour

I am getting the disconnection even though I am subscribed to some pairs and when I try to subscribe again, I get an WebSocket is already in CLOSING or CLOSED state. error in the browser console.

Why does the websocket connection with the gateway close?

[aloysius-pgast]

You need to re-create a Websocket after ws is disconnected in the browser

[lolero]

Ok. But why does the socket close all of a sudden while there are active pair subscriptions?

[aloysius-pgast]

Could you post gateway log (running with DEBUG) and content of onclose event in browser ?

[lolero]

Sure! How do I run the gateway with DEBUG?

[aloysius-pgast]

Try the following (both should be done) :

• set "loglevel":"debug" in config.json
• run gateway using DEBUG=CEG:Session node gateway.js

[lolero]

I there way to do it using the docker container?

[aloysius-pgast]

Yes, just define both environment variables :

• cfg.logLevel=debug
• DEBUG=CEG:Session

[lolero]

Here are the logs. As you can see, the gateway is working fine and receiving order book updates normally and then all of a sudden something happens and it begins to disconnect :/

2020-05-25T06:30:02.678Z CEG:Session Received 'orderBookUpdate' event from exchange 'bittrex' for pair 'BTC-OCEAN' : {"cseq":153026,"buySize":0,"sellSize":3}
2020-05-25T06:30:02.681Z CEG:Session Unregistering socket '1' (172.17.0.1) from session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f' : 0 sockets remaining
2020-05-25T06:30:02.681Z CEG:Session Removing listeners for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
2020-05-25T06:30:02.681Z CEG:Session Removing 'ticker (bittrex)' listener for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
2020-05-25T06:30:02.681Z CEG:Session Removing 'orderBook (bittrex)' listener for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
2020-05-25T06:30:02.681Z CEG:Session Removing 'orderBookUpdate (bittrex)' listener for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
2020-05-25T06:30:02.681Z CEG:Session Removing 'trades (bittrex)' listener for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
2020-05-25T06:30:02.682Z CEG:Session Removing 'kline (bittrex)' listener for session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f'
1590388202689|INFO|Connection (bittrex|4) will be disconnected
2020-05-25T06:30:02.700Z CEG:Session Session 'rpc.bf099f02-45a9-4301-8ec9-b194f62f2c3f' will be destroyed in 600s

Here is the close event from the onclose callback:

bubbles: false
cancelBubble: false
cancelable: false
code: 1006
composed: false
currentTarget: WebSocket
binaryType: "blob"
bufferedAmount: 0
extensions: ""
onclose: event => {…}
onerror: err => console.error('error:', err)
onmessage: event => {…}
onopen: null
protocol: ""
readyState: 3
url: "ws://127.0.0.1:8101/"
__proto__: WebSocket
defaultPrevented: false
eventPhase: 0
isTrusted: true
path: []
reason: ""
returnValue: true
srcElement: WebSocket
binaryType: "blob"
bufferedAmount: 0
extensions: ""
onclose: event => {…}
onerror: err => console.error('error:', err)
onmessage: event => {…}
onopen: null
protocol: ""
readyState: 3
url: "ws://127.0.0.1:8101/"
__proto__: WebSocket
target: WebSocket
binaryType: "blob"
bufferedAmount: 0
extensions: ""
onclose: event => {…}
onerror: err => console.error('error:', err)
onmessage: event => {…}
onopen: null
protocol: ""
readyState: 3
url: "ws://127.0.0.1:8101/"
__proto__: WebSocket
timeStamp: 236145.6249999901
type: "close"
wasClean: false

[aloysius-pgast]

Did you also set environment variable cfg.logLevel=debug ? Because I don't see any log line with |DEBUG|

[lolero]

Here is my docker command:

docker run --rm -p 8100:8000 -p 8101:8001 --name crypto-exchanges-gateway -e cfg.exchanges.bittrex.key='xxx' -e cfg.exchanges.bittrex.secret='xxx' -e cfg.logLevel=debug -e DEBUG=CEG:Session apendergast/crypto-exchanges-gateway

[lolero]

And here is the output without DEBUG=CEG:Session

1590397315461|WARN|Starting...
1590397315478|WARN|MarketCap API is enabled
1590397315478|WARN|FxConverter API is enabled
1590397315479|WARN|poloniex exchange (poloniex) is enabled (public API)
1590397315479|WARN|bittrex exchange (bittrex) is enabled (public API & trading API)
1590397315479|WARN|binance exchange (binance) is enabled (public API)
1590397315479|WARN|kucoin exchange (kucoin) is enabled (public API)
1590397315480|WARN|okex exchange (okex) is enabled (public API)
1590397315480|WARN|UI is enabled
1590397315480|WARN|TickerMonitor is enabled
1590397316268|INFO|Database is ok
1590397316273|INFO|0 sessions loaded
1590397316274|INFO|0 TickerMonitor entries loaded
1590397316281|INFO|Data loaded successfully
1590397316302|WARN|HTTP server is alive on *:8000
1590397316303|WARN|WS server is alive on *:8001
1590397319217|INFO|Got new connection for RPC session 'rpc.cab862db-035f-4d79-9e37-01b4d56e5bf1' : ipaddr = '172.17.0.1'
1590397319811|DEBUG|172.17.0.1 GET /exchanges/bittrex/pairs
1590397320164|DEBUG|172.17.0.1 GET /exchanges/bittrex/tickers
1590397324785|DEBUG|172.17.0.1 GET /exchanges/bittrex/balances
1590397340855|INFO|Connection (bittrex|1|bbebb563-8a43-4107-8188-0c2c92921b6f) connected
1590397439224|DEBUG|Got timeout for WS rpc.cab862db-035f-4d79-9e37-01b4d56e5bf1/1
1590397439258|INFO|Connection (bittrex|1) will be disconnected

For some reason the timeout wasn't there last time I posted it. What causes that timeout?

[aloysius-pgast]

The websocket is disconnected because it didn't answer to ws ping sent by the gateway, exactly 2 minutes after successful connection (Got timeout for WS rpc.cab862db-035f-4d79-9e37-01b4d56e5bf1/1)

[aloysius-pgast]

ws ping/pong is usually handled directly by the WebSocket class in browser AFAIK. Which browser are you using ?

[lolero]

I use Chrome.

[aloysius-pgast]

And how to you open the ws connection ?

[lolero]

const ws = new WebSocket('ws://127.0.0.1:8101');

[aloysius-pgast]

Could you try to create an html file with below content, check what you get in chrome console and see if you have an indication that ws was terminated after some time ?

<html>
<head>
<script>
document.addEventListener('DOMContentLoaded', () => {

const wsUri = 'ws://127.0.0.1:8101';

const ws = new WebSocket(wsUri);

let connectedTimestamp = Date.now();
let closedTimestamp = Date.now();

ws.onopen = (e) => {
    connectedTimestamp = Date.now();
    console.log(`${new Date().toISOString()} - ws successfully connected`);
}

ws.onerror = (e) =>  {
    closedTimestamp = Date.now();
    const delay = (closedTimestamp - connectedTimestamp);
    console.error(`${new Date().toISOString()} - Could not open ws connection to '${wsUri}' (delay = ${delay}ms)`);
}

ws.onclose = (e) =>  {
    closedTimestamp = Date.now();
    const delay = (closedTimestamp - connectedTimestamp);
    console.error(`${new Date().toISOString()} - ws connection '${wsUri}' was closed (code = '${e.code}'', reason = '${e.reason}') (delay = ${delay}ms)`);    
}

ws.onmessage = (e) => {
    console.log(`${new Date().toISOString()} - Got ws message from '${wsUri}'`);
    console.log(e.data);
}

});
</script>
</head>
<body>
</body>
</html>

[lolero]

No disconnection :/

2020-05-25T11:24:53.127Z - ws successfully connected
2020-05-25T11:24:53.141Z - Got ws message from 'ws://127.0.0.1:8101'
{"hello":{"sid":"rpc.f580c347-2417-4884-b24e-4bb1f9482e1a","isNew":true}}

I guess its a problem with my implementation.

Ill post back if I figure out whats wrong. Thank you so very much for all the help and support, though!!

[lolero]

It turns out that the issue is a function of how many pairs I am subscribed to and happens only on chrome. It would seem like after so many processing threads initiated by different threads, chrome gets overloaded and fails to respond to the gateway's ping :/

Either way, it doesn't happen on firefox so I will continue working there.