search

ctubio / tribeca

Self-hosted crypto trading bot (automated high frequency market making) in node.js, angular, typescript and c++
https://127.0.0.1:3000
Other
95 stars 26 forks source link

(BUG) Stop sending orders in Bitfinex #122

Closed Camille92 closed 7 years ago

Camille92 commented 7 years ago

Hello Carles,

I was suspecting this bug to happen before but I didn't the opportunity to see it in action.

Somehow Tribeca in some pairs in Bitfinex stops sending trades after some time.

capture d ecran 2017-06-05 a 16 24 16

As you can see here, altough Tribeca BTC/USD is up and running it did not send any trade since 3am (it 4pm time of writing). capture d ecran 2017-06-05 a 16 25 34

It says it's sending orders from the market but I can't find any sign of that.

Any idea what's happening and how I can help you to fix that ?

Best,

Camille :)

ctubio commented 7 years ago

im sorry im not running Bitfinex anymore because of the fees, Bitfinex already eat my 0.02 testing funds :( and i dont have patience to set a profitable strategy while testing (while testing i don't use any trading strategy just place blind orders to whatever price to be able to test whatever is needed)

i would need or more funds or more patience or a testing account :P will keep the issue open in case anyone wanna provide me any of these

thanks for your report'¡

Camille92 commented 7 years ago

No problem I send you that! (and probably a bit more, but don't burn everything in fees :p )

I'm running some tests from my side in order to understand the issue. Maybe Bitfinex can't manage to receive orders in all markets at the same time, maybe it's something else :/ but here are some logs I managed to have.

nodejs tribeca.js bitdashbtc.json

16:04:33.226 main Requiring authentication to web client
16:04:33.250 main Listening to admins on *: 3014
16:04:33.490 main Unhandled promise rejection! TypeError: symbolDetails[Symbol.iterator] is not a function
    at Object.<anonymous> (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:475:23)
    at Generator.next (<anonymous>)
    at fulfilled (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:4:58)
    at tryCatcher (/home/ubuntu/tribeca/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5) Promise {
  _bitField: 18087936,
  _fulfillmentHandler0: 
   TypeError: symbolDetails[Symbol.iterator] is not a function
       at Object.<anonymous> (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:475:23)
       at Generator.next (<anonymous>)
       at fulfilled (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:4:58)
       at tryCatcher (/home/ubuntu/tribeca/node_modules/bluebird/js/release/util.js:16:23)
       at Promise._settlePromiseFromHandler (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:512:31)
       at Promise._settlePromise (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:569:18)
       at Promise._settlePromise0 (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:614:10)
       at Promise._settlePromises (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:693:18)
       at Async._drainQueue (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:133:16)
       at Async._drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:143:10)
       at Immediate.Async.drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:17:14)
       at runCallback (timers.js:672:20)
       at tryOnImmediate (timers.js:645:5)
       at processImmediate [as _immediateCallback] (timers.js:617:5),
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
16:04:35.496 main Did not complete clean-up tasks successfully, still shutting down.
16:04:35.496 main Exiting with code 1
ctubio commented 7 years ago

:sunflower: TX received!

ctubio commented 7 years ago

how many concurrent instances of tribeca (diff currency pairs) are you using against Bitfinex? i will try somewhat the same amount

Camille92 commented 7 years ago

15 instances ^^' but I barely never have more than 10 or 11 orders on the market (when it should be around 15 or 20)

Basically all the pairs BTC, ETH, ETC, ZEC, XMR, LTC, DASH & XRP

ctubio commented 7 years ago

that error you pasted means bitfinex did not respond to https://api.bitfinex.com/v1/symbols_details, we could save the response in a cached file so we dont need to ask on every boot lets say with a daily refresh; but then next call like pubticker or buy/sell will mostly possible fail too.

the solution here is to centralize our communications so we dont open 15 connections that independently will block each other. << this will arrive once we have child processes for gateway logics, so multiple tribecas may connecto to the same child process without spawn cloned logic (im still wondering if use https://github.com/substack/dnode or to directly move all gateways to c++;but this is make time several time' :dancer:)

im not sure if there is anything else to fix here, so do you agree if we consider your $$$ sent as towards the implementation of child processes for gateways? (it make take several time but i think is a must at some point)

Camille92 commented 7 years ago

Well, the weird thing about that is that when I tried a second time it worked perfectly! So maybe it's not responding at one moment because it's occupied by other orders.

Yes for sure I was thinking about the necessity of this one as well, not for the same reason but achieving both is best. I just write that here quickly but:

The idea is to use this child process as a "manager" for TBP, because now we can determine what's a good coin to bet on but not what's the BEST coin to bet for. Ie: If both ETH and BTC are going up by TBP position will be for both of them Full crypto, even though BTC might be up by 2% and ETH 10%. We would need this process to have an overview of the different market and defines a good TBP for each depending on the results of the others markets !

ctubio commented 7 years ago

ok well, back to the present (cos this will take several time), maybe we can implement some retry if a call fails, cos as you said seems to work after a little waiting time (but if you run with forever, this retry should happen auto)

Camille92 commented 7 years ago

I'll keep working on that issue because I don't have the logs when it goes from a situation of trading to a situation of not trading.

Maybe by opening 15 terminal and waiting to see where something happens!

Yes the retry should be a good solution :)

ctubio commented 7 years ago

ok ty, will try to retry failed calls

Camille92 commented 7 years ago

Hello Carles,

After doing some testing and opening all the Tribeca in the console to see what's happening.

Here is what I can say:

1) Bug solution

I think, but I'm not 100% sure, that the bug comes simply by the fact that the instances are "fighting" for the same amount of cash / BTC and some of them can't send orders because it's too big.

I was thinking about a solution in quoting engine, it should ease or solve the problem:

If Sell.OrderSize > latestPosition.baseAmout / 2
Sell.OrderSize = min.Size
If Buy.OrderSize > (latestPosition.quoteAmount / fv.price) / 2
Buy.OrderSize = min.Size

Doing so we reduce the size of orders to a minimum only if the cash (or btc) we have in the account is limited and that order would represent over 50% of what's left.

What do you think about it? :)

2) Some logs I got (always interesting)

Here are some logs on Bitfinex, I know it's not the topic but it is interesting to keep them somewhere in case we need them in the future:

A) The "I don't wanna boot": -> Symbol.iterator is not a function ?

12:17:49.582 main Requiring authentication to web client
12:17:49.605 main Listening to admins on *: 3014
12:17:49.764 main Unhandled promise rejection! TypeError: symbolDetails[Symbol.iterator] is not a function
    at Object.<anonymous> (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:475:23)
    at Generator.next (<anonymous>)
    at fulfilled (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:4:58)
    at tryCatcher (/home/ubuntu/tribeca/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5) Promise {
  _bitField: 18087936,
  _fulfillmentHandler0: 
   TypeError: symbolDetails[Symbol.iterator] is not a function
       at Object.<anonymous> (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:475:23)
       at Generator.next (<anonymous>)
       at fulfilled (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:4:58)
       at tryCatcher (/home/ubuntu/tribeca/node_modules/bluebird/js/release/util.js:16:23)
       at Promise._settlePromiseFromHandler (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:512:31)
       at Promise._settlePromise (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:569:18)
       at Promise._settlePromise0 (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:614:10)
       at Promise._settlePromises (/home/ubuntu/tribeca/node_modules/bluebird/js/release/promise.js:693:18)
       at Async._drainQueue (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:133:16)
       at Async._drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:143:10)
       at Immediate.Async.drainQueues (/home/ubuntu/tribeca/node_modules/bluebird/js/release/async.js:17:14)
       at runCallback (timers.js:672:20)
       at tryOnImmediate (timers.js:645:5)
       at processImmediate [as _immediateCallback] (timers.js:617:5),
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
12:17:51.770 main Did not complete clean-up tasks successfully, still shutting down.
12:17:51.771 main Exiting with code 1

B) The "CloudFlare strike": Possible solution ? -> Getting market data from the WebSocket API ?

I get this one a lot ^^"

13:04:35.330 bitfinex SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at Request.request [as _callback] (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:394:41)
    at Request.self.callback (/home/ubuntu/tribeca/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:194:7)
    at Request.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:191:7)
    at IncomingMessage.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1091:12)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:975:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9) Error parsing JSON url= https://api.bitfinex.com/v1/balances err= SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at Request.request [as _callback] (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:394:41)
    at Request.self.callback (/home/ubuntu/tribeca/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:194:7)
    at Request.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:191:7)
    at IncomingMessage.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1091:12)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:975:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9) , body= <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>

<meta http-equiv="set-cookie" content="cf_use_ob=0; expires=Tue, 06-Jun-17 13:05:05 GMT; path=/">

<title>api.bitfinex.com | 520: Web server is returning an unknown error</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
<!--[if lte IE 9]><script type="text/javascript" src="/cdn-cgi/scripts/jquery.min.js"></script><![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script>

</head>
<body>
<div id="cf-wrapper">

    <div id="cf-error-details" class="cf-error-details-wrapper">
        <div class="cf-wrapper cf-error-overview">
            <h1>
              <span class="cf-error-type" data-translate="error">Error</span>
              <span class="cf-error-code">520</span>
              <small class="heading-ray-id">Ray ID: 36ab940b1b1923b4 &bull; 2017-06-06 13:04:35 UTC</small>
            </h1>
            <h2 class="cf-subheadline" data-translate="error_desc">Web server is returning an unknown error</h2>
        </div><!-- /.error-overview -->

        <div class="cf-section cf-highlight cf-status-display">
            <div class="cf-wrapper">
                <div class="cf-columns cols-3">

<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc" data-translate="browser_desc">You</span>
  <h3 class="cf-status-name" data-translate="browser_label">Browser</h3>
  <span class="cf-status-label" data-translate="browser_status_label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc" data-translate="cloud_desc">Ashburn</span>
  <h3 class="cf-status-name" data-translate="cloud_label">Cloudflare</h3>
  <span class="cf-status-label" data-translate="cloud_status_label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc" data-translate="server_desc">api.bitfinex.com</span>
  <h3 class="cf-status-name" data-translate="server_label">Host</h3>
  <span class="cf-status-label" data-translate="server_status_label">Error</span>
</div>

                </div>
            </div>
        </div><!-- /.status-display -->

        <div class="cf-section cf-wrapper">
            <div class="cf-columns two">
                <div class="cf-column">
                    <h2 data-translate="what_happened">What happened?</h2>
                    <p data-translate="unknown_error_desc">There is an unknown connection issue between Cloudflare and the origin web server. As a result, the web page can not be displayed.</p>
                </div>

                <div class="cf-column">
                    <h2 data-translate="what_can_i_do">What can I do?</h2>
                          <h5 data-translate="if_website_visitor">If you are a visitor of this website:</h5>
      <p data-translate="try_again_in_a_few">Please try again in a few minutes.</p>

      <h5 data-translate="if_website_owner">If you are the owner of this website:</h5>
      <p><span data-translate="520_owner_desc">There is an issue between Cloudflare's cache and your origin web server. Cloudflare monitors for these errors and automatically investigates the cause. To help support the investigation, you can pull the corresponding error log from your web server and submit it our support team.  Please include the Ray ID (which is at the bottom of this error page).</span> <a href="https://support.cloudflare.com/hc/en-us/articles/200171936-Error-520" data-translate="addtl_troubleshooting">Additional troubleshooting resources</a>.</p>
                </div>
            </div>
        </div><!-- /.section -->

        <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>36ab940b1b1923b4</strong></span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span data-translate="your_ip">Your IP</span>: 54.226.225.18</span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span data-translate="performance_security_by">Performance &amp; security by</span> <a data-orig-proto="https" data-orig-ref="www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->

    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->

13:04:35.334 main Unhandled promise rejection! SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at Request.request [as _callback] (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:394:41)
    at Request.self.callback (/home/ubuntu/tribeca/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:194:7)
    at Request.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:191:7)
    at IncomingMessage.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1091:12)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:975:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9) Promise {
  _bitField: 18087936,
  _fulfillmentHandler0: 
   SyntaxError: Unexpected token < in JSON at position 0
       at JSON.parse (<anonymous>)
       at Request.request [as _callback] (/home/ubuntu/tribeca/app/server/gateways/bitfinex.js:394:41)
       at Request.self.callback (/home/ubuntu/tribeca/node_modules/request/request.js:188:22)
       at emitTwo (events.js:106:13)
       at Request.emit (events.js:194:7)
       at Request.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1171:10)
       at emitOne (events.js:96:13)
       at Request.emit (events.js:191:7)
       at IncomingMessage.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:1091:12)
       at Object.onceWrapper (events.js:293:19)
       at emitNone (events.js:91:20)
       at IncomingMessage.emit (events.js:188:7)
       at endReadableNT (_stream_readable.js:975:12)
       at _combinedTickCallback (internal/process/next_tick.js:80:11)
       at process._tickCallback (internal/process/next_tick.js:104:9),
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
13:04:35.339 main All exiting event handlers have fired, exiting application.
13:04:35.339 main Exiting with code 0

C) The "Esockettimedout"

12:27:19.312 bitfinex { Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:819:19)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:188:7)
    at TLSSocket.emitTimeout (_http_client.js:679:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at TLSSocket.emit (events.js:188:7)
    at TLSSocket.Socket._onTimeout (net.js:352:8)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5) code: 'ESOCKETTIMEDOUT', connect: false } Error returned: url= https://api.bitfinex.com/v1/order/cancel err= { Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:819:19)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:188:7)
    at TLSSocket.emitTimeout (_http_client.js:679:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at TLSSocket.emit (events.js:188:7)
    at TLSSocket.Socket._onTimeout (net.js:352:8)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5) code: 'ESOCKETTIMEDOUT', connect: false }
12:27:19.316 main Unhandled promise rejection! { Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:819:19)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:188:7)
    at TLSSocket.emitTimeout (_http_client.js:679:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at TLSSocket.emit (events.js:188:7)
    at TLSSocket.Socket._onTimeout (net.js:352:8)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5) code: 'ESOCKETTIMEDOUT', connect: false } Promise {
  _bitField: 18087936,
  _fulfillmentHandler0: 
   { Error: ESOCKETTIMEDOUT
       at ClientRequest.<anonymous> (/home/ubuntu/tribeca/node_modules/request/request.js:819:19)
       at Object.onceWrapper (events.js:293:19)
       at emitNone (events.js:86:13)
       at ClientRequest.emit (events.js:188:7)
       at TLSSocket.emitTimeout (_http_client.js:679:10)
       at Object.onceWrapper (events.js:293:19)
       at emitNone (events.js:86:13)
       at TLSSocket.emit (events.js:188:7)
       at TLSSocket.Socket._onTimeout (net.js:352:8)
       at ontimeout (timers.js:386:14)
       at tryOnTimeout (timers.js:250:5)
       at Timer.listOnTimeout (timers.js:214:5) code: 'ESOCKETTIMEDOUT', connect: false },
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
12:27:19.320 main All exiting event handlers have fired, exiting application.
12:27:19.320 main Exiting with code 0
Camille92 commented 7 years ago

Erratum the bug is actually something different.

ctubio commented 7 years ago

im not sure that i understand your last message :sob:

Camille92 commented 7 years ago

Haha in the previous message I was saying that maybe there is no bug, just Tribeca not sending orders because there is not enough money left to trade with, and because I was not able to see the bug in action with an eventual output in the console.

In the last one, I come back on what I've said because I've seen the bug in action (but not while I was looking at the logs).

Camille92 commented 7 years ago

Heyyy after some time running Tribeca and registering the logs, I've finally caught the bug I was talking about and describing here.

(On the ETC USD market)

Apparently, it's a lost of heartbeat and then not a good reconnection to the server.

Do you think you can fix it ? 😄 


21:10:06.894 bitfinex Heartbeat lost, reconnecting...
21:10:06.992 active Changed quoting mode to true
21:10:06.999 bitfinex Bitfinex info: { event: 'info', version: 2 }
21:10:07.099 bitfinex Successfully connected to trades
21:10:07.099 bitfinex Got message on unknown topic [ 326,
  [ [ 36048513, 1496783320000, 171.224923, 17 ],
    [ 36048489, 1496783264000, 3.7938, 17 ],
    [ 36048488, 1496783264000, 0.5, 16.984 ],
    [ 36048487, 1496783264000, 0.5, 16.984 ],
    [ 36048420, 1496783144000, 0.5, 16.964 ],
    [ 36048393, 1496783110000, 41.903267, 16.992 ],
    [ 36048392, 1496783110000, 39.0404, 16.986 ],
    [ 36048379, 1496783099000, 31.415048, 16.98 ],
    [ 36048376, 1496783095000, 10.024997, 16.985 ],
    [ 36048364, 1496783077000, 51.09699704, 16.912 ],
    [ 36048361, 1496783076000, 7.902453, 16.912 ],
    [ 36048358, 1496783072000, 0.88412859, 16.912 ],
    [ 36048355, 1496783072000, 45.802421, 16.912 ],
    [ 36048354, 1496783072000, 45.802421, 16.912 ],
    [ 36048353, 1496783071000, 43.828332, 16.912 ],
    [ 36048352, 1496783071000, -1, 16.918 ],
    [ 36048351, 1496783071000, -1, 16.918 ],
    [ 36048350, 1496783071000, -0.5, 16.925 ],
    [ 36048349, 1496783071000, -0.5, 16.925 ],
    [ 36048348, 1496783071000, -0.5, 16.926 ],
    [ 36048347, 1496783071000, -0.15178275, 16.927 ],
    [ 36048346, 1496783071000, -0.1506642, 16.928 ],
    [ 36048345, 1496783071000, -0.15078365, 16.929 ],
    [ 36048344, 1496783071000, -0.15135337, 16.93 ],
    [ 36048343, 1496783070000, -0.15224168, 16.931 ],
    [ 36048342, 1496783070000, -0.27338447, 16.932 ],
    [ 36048341, 1496783070000, -0.15303725, 16.933 ],
    [ 36048337, 1496783060000, 27.840804, 16.943 ],
    [ 36048332, 1496783048000, -0.15430004, 16.96 ],
    [ 36048331, 1496783048000, -0.5, 16.96 ] ] ]
21:10:07.124 bitfinex Authentication status: OK
21:10:07.702 bitfinex Successfully connected to book
21:10:07.702 bitfinex Got message on unknown topic [ 27754,
21:10:07.702 bitfinex Got message on unknown topic [ 27754,
  [ [ 16.956, 1, 0.14982802 ],
    [ 16.955, 1, 0.15000685 ],
    [ 16.954, 1, 0.38556069 ],
    [ 16.953, 1, 0.15044675 ],
    [ 16.952, 1, 0.15465573 ],
    [ 16.951, 3, 194.6100357 ],
    [ 16.95, 1, 0.15113046 ],
    [ 16.949, 1, 0.15442234 ],
    [ 16.948, 1, 0.21196263 ],
    [ 16.947, 1, 0.1481466 ],
    [ 16.946, 1, 0.15054224 ],
    [ 16.945, 1, 0.15138133 ],
    [ 16.944, 2, 1 ],
    [ 16.915, 1, 4.49177678 ],
    [ 16.914, 2, 333.19021013 ],
    [ 16.905, 1, 0.15406575 ],
    [ 16.904, 1, 0.1522587 ],
    [ 16.903, 1, 0.56429711 ],
    [ 16.902, 3, 1146.1591 ],
    [ 16.849, 3, 2.53 ],
    [ 16.836, 1, 0.5 ],
    [ 16.83, 2, 1 ],
    [ 16.829, 1, 0.5 ],
    [ 16.826, 1, 1 ],
    [ 16.821, 1, 1 ],
    [ 17, 11, -1649.97430469 ],
    [ 17.004, 1, -0.1 ],
    [ 17.006, 1, -0.5 ],
    [ 17.007, 1, -1 ],
    [ 17.008, 1, -0.5 ],
    [ 17.01, 1, -0.5 ],
21:10:07.704 bitfinex Got message on unknown topic [ 27754, [ 16.821, 0, 1 ] ]
21:10:07.704 bitfinex Got message on unknown topic [ 27754, [ 16.856, 1, 271.51969609 ] ]
21:10:07.704 bitfinex Got message on unknown topic [ 27754, [ 17.038, 2, -1.14013843 ] ]
21:10:09.844 bitfinex Got message on unknown topic [ 27754, [ 16.856, 0, 1 ] ]
21:10:09.844 bitfinex Got message on unknown topic [ 27754, [ 16.821, 1, 1 ] ]
21:10:10.133 bitfinex Got message on unknown topic [ 27754, [ 16.821, 0, 1 ] ]
21:10:10.133 bitfinex Got message on unknown topic [ 27754, [ 16.85, 1, 271.61637966 ] ]
21:10:10.606 bitfinex Got message on unknown topic [ 27754, [ 17.02, 3, -1.5 ] ]
21:10:10.893 bitfinex Got message on unknown topic [ 27754, [ 16.826, 0, 1 ] ]
21:10:10.893 bitfinex Got message on unknown topic [ 27754, [ 16.916, 1, 138.68223854 ] ]
21:10:10.893 bitfinex Got message on unknown topic [ 27754, [ 16.914, 1, 194.5055 ] ]
21:10:10.893 bitfinex Got message on unknown topic [ 27754, [ 17.02, 4, -7.23187373 ] ]
21:10:11.621 bitfinex Got message on unknown topic [ 27754, [ 16.915, 0, 1 ] ]
21:10:11.621 bitfinex Got message on unknown topic [ 27754, [ 16.914, 0, 1 ] ]
21:10:11.621 bitfinex Got message on unknown topic [ 27754, [ 16.917, 1, 4.49124574 ] ]
21:10:11.621 bitfinex Got message on unknown topic [ 27754, [ 16.916, 2, 333.21243854 ] ]
21:10:11.622 bitfinex Got message on unknown topic [ 27754, [ 16.826, 1, 1 ] ]
21:10:15.165 bitfinex Got message on unknown topic [ 27754, [ 17.022, 0, -1 ] ]
21:10:15.166 bitfinex Got message on unknown topic [ 27754, [ 17.053, 1, -0.13986986 ] ]
21:10:21.318 bitfinex Got message on unknown topic [ 27754, [ 17.024, 1, -0.1 ] ]
21:10:22.445 bitfinex Got message on unknown topic [ 27754, [ 17.02, 3, -1.5 ] ]
21:10:22.698 bitfinex Got message on unknown topic [ 27754, [ 16.826, 0, 1 ] ]
21:10:22.698 bitfinex Got message on unknown topic [ 27754, [ 16.918, 1, 138.67976704 ] ]
21:10:22.698 bitfinex Got message on unknown topic [ 27754, [ 16.916, 1, 194.5302 ] ]
21:10:22.699 bitfinex Got message on unknown topic [ 27754, [ 17.053, 0, -1 ] ]
21:10:22.699 bitfinex Got message on unknown topic [ 27754, [ 17.022, 1, -5.73187373 ] ]
21:10:22.951 bitfinex Got message on unknown topic [ 27754, [ 16.916, 0, 1 ] ]
21:10:22.951 bitfinex Got message on unknown topic [ 27754, [ 16.826, 1, 1 ] ]
21:10:22.951 bitfinex Got message on unknown topic [ 27754, [ 17.052, 0, -1 ] ]
21:10:22.951 bitfinex Got message on unknown topic [ 27754, [ 17.039, 1, -0.14233209 ] ]
21:10:23.204 bitfinex Got message on unknown topic [ 27754, [ 16.917, 0, 1 ] ]
21:10:23.204 bitfinex Got message on unknown topic [ 27754, [ 16.918, 2, 333.22606704 ] ]
21:10:23.204 bitfinex Got message on unknown topic [ 27754, [ 16.821, 1, 1 ] ]
21:11:19.426 tbp recalculated 32.343075853011726

Then it keeps going on like that until a restart

Camille92 commented 7 years ago

Update,

I've found a solution to the issue. I created more API keys, 1 key for 2 or 3 markets and now I don't lose Heartbeat. :D

So sure it's not a fix, but at least the situation doesn't happen anymore.