ctubio / tribeca

Self-hosted crypto trading bot (automated high frequency market making) in node.js, angular, typescript and c++
https://127.0.0.1:3000
Other
95 stars 26 forks source link

Coinbase Stack Trace: Cannot read property 'size' of undefined at getQuoteAtTopOfMarket #98

Closed zachaller closed 7 years ago

zachaller commented 7 years ago

I have been having a lot of random dissconnects where i can not re connect to coinbase I hit the red button and nothing will happen cant keep it running for more than maybe 2-3 hrs. I do not know if this is related but I also started running tribeca not in forever to see if I could get some more logging which also showed this error and crash

main Unhandled exception! TypeError: Cannot read property 'size' of undefined
    at Object.getQuoteAtTopOfMarket (/home/zaller/tribeca/app/server/quoting-styles/helpers.js:28:39)
    at computeBoomerangQuote (/home/zaller/tribeca/app/server/quoting-styles/ping-pong.js:65:30)
    at BoomerangQuoteStyle.GenerateQuote (/home/zaller/tribeca/app/server/quoting-styles/ping-pong.js:18:20)
    at QuotingEngine.computeQuote (/home/zaller/tribeca/app/server/quoting-engine.js:106:59)
    at Array.QuotingEngine.recalcQuote (/home/zaller/tribeca/app/server/quoting-engine.js:62:32)
    at Evt.trigger (/home/zaller/tribeca/app/server/utils.js:31:43)
    at MarketFiltration.set latestFilteredMarket [as latestFilteredMarket] (/home/zaller/tribeca/app/server/market-filtration.js:44:36)
    at Evt.MarketFiltration.filterFullMarket [as _singleCallback] (/home/zaller/tribeca/app/server/market-filtration.js:20:39)
    at Evt.trigger (/home/zaller/tribeca/app/server/utils.js:28:22)
    at Evt.MarketDataBroker.handleMarketData [as _singleCallback] (/home/zaller/tribeca/app/server/broker.js:24:29)
    at Evt.trigger (/home/zaller/tribeca/app/server/utils.js:28:22)
    at CoinbaseMarketDataGateway.raiseMarketData (/home/zaller/tribeca/app/server/gateways/coinbase.js:74:33)
    at CoinbaseMarketDataGateway.onOrderBookChanged (/home/zaller/tribeca/app/server/gateways/coinbase.js:60:18)
    at CoinbaseMarketDataGateway.onMessage (/home/zaller/tribeca/app/server/gateways/coinbase.js:29:22)
    at OrderbookSync.CoinbaseMarketDataGateway._client.on.data (/home/zaller/tribeca/app/server/gateways/coinbase.js:79:49)
    at emitOne (events.js:101:20)
    at OrderbookSync.emit (events.js:191:7)
main All exiting event handlers have fired, exiting application.
main Exiting with code 0
Camille92 commented 7 years ago

Same problem here.

I evoked a solution in #90 to restart Tribeca if we get disconnected from server.

What do you think @ctubio ?

ctubio commented 7 years ago

yea this happens since using https://github.com/coinbase/gdax-node, let me try to fix it somehow without rolling back to our previous custom integration

:boom: seems like market data sometimes is empty in coinbase (<<situation never expected and never happening with the old custom integration)

ambethia commented 7 years ago

Is this why my bots keep shutting off after a few hours?

ctubio commented 7 years ago

@ambethia i bet 1 coin it is

ctubio commented 7 years ago
Trace: active Changed quoting mode to false
    at ActiveRepository.updateParameters (/home/admin/tribeca/app/server/active-state.js:30:25)
    at Evt.ActiveRepository._exchangeConnectivity.ConnectChanged.on [as _singleCallback] (/home/admin/tribeca/app/server/active-state.js:39:60)
    at Evt.trigger (/home/admin/tribeca/app/server/utils.js:28:22)
    at ExchangeBroker.onConnect (/home/admin/tribeca/app/server/broker.js:519:33)
    at Evt.ExchangeBroker._oeGateway.ConnectChanged.on.s [as _singleCallback] (/home/admin/tribeca/app/server/broker.js:527:18)
    at Evt.trigger (/home/admin/tribeca/app/server/utils.js:28:22)
    at CoinbaseOrderEntryGateway.onStateChange (/home/admin/tribeca/app/server/gateways/coinbase.js:214:33)
    at OrderbookSync.CoinbaseOrderEntryGateway._client.on.data (/home/admin/tribeca/app/server/gateways/coinbase.js:286:47)
    at emitNone (events.js:91:20)
    at OrderbookSync.emit (events.js:185:7)
    at OrderbookSync.prototype.onClose (/home/admin/tribeca/node_modules/gdax/lib/clients/websocket.js:86:10)
    at emitTwo (events.js:106:13)
    at WebSocket.emit (events.js:191:7)
    at WebSocket.cleanupWebsocketResources (/home/admin/tribeca/node_modules/ws/lib/WebSocket.js:950:8)
    at emitNone (events.js:91:20)
    at TLSSocket.emit (events.js:185:7)
ctubio commented 7 years ago

im about to recover the old integration, cos didnt lost connection, but had market data messed up (i preffer to fix the market data again than keep fighting with this connection lost bug of https://github.com/coinbase/gdax-node/issues/24 xD)

zachaller commented 7 years ago

Ok still testing this but I would say so far it has improved a lot but time will tell also not sure if this is related or not but I am still getting a build up of not canceled orders even with cxl on which i would expect every 5min to cancel all open orders.

example

ctubio commented 7 years ago

im not seen zombie orders in Coinbase since long ago but is true that the 5min cancel is not working always in Coinbase, i do not know why, it works always on boot and sometimes after too (if you are lucky). i will investigate the cxl feature soOn

if you get every day zombie orders feel free to leave a msg cos im not looking into that since long ago

many thanks'¡

zachaller commented 7 years ago

Strange I get so many zombie orders that it is almost unusable they build up within 30min to an hour I can only run it while i watch it because I have to go into coinbase web ui and hit the cancel all orders button every hr or so.

zachaller commented 7 years ago

should this not return true https://github.com/ctubio/tribeca/blob/master/src/server/gateways/coinbase.ts#L156

ctubio commented 7 years ago

this is OK with false, note that this only is a flag to denote if the function cancelAllOpenOrders should be called (defined just in the line under), but no matter if the flag is true or false, cancelAllOpenOrders method is efectively a public method, so: at https://github.com/ctubio/tribeca/blob/master/src/server/broker.ts#L44 (when clickin the Cancel Orders button in the UI) is only called depending on the flag (so the UI button of Cancel Orders only cancels orders in tribeca, not orders in the exchange) at https://github.com/ctubio/tribeca/blob/master/src/server/broker.ts#L458 (every 5min) is called always without matter the flag (so every 5min both tribeca and exchange trades are canceled) among other places where it is also used

as true will make the UI button of Cancel Orders to not only cancel tribeca orders but also orders in the exchange; has no effect for the 5min cxl feature cos is not based on the flag

maybe instead of supportsCancelAllOpenOrders should be called uiButtonCancelAllOpenOrdersInExchange

hope this helps¡'

ctubio commented 7 years ago

issue fixed'¡

00:17:16.216 active Changed quoting mode to false
00:17:21.275 active Changed quoting mode to true

(all auto)