GaloyMoney / dealer

79 stars 19 forks source link

timeout: increase latency #139

Open nicolasburtey opened 2 years ago

nicolasburtey commented 2 years ago

there is a lot of timeout and related alert. increasing latency (time for the client to get a response) may help.

cc @ntheile

ntheile commented 2 years ago

@sebastienverreault

Here is a link to the HTTP trace that takes about 28 seconds to complete. https://ui.honeycomb.io/galoy/datasets/galoy-staging/result/HQiS4jQr8Vu/trace/DJT7RQJDRrC?fields[]=c_name&fields[]=c_service.name&span=bedb8bee50d4762b . This appears to be alerting about 10 times each hour.

It looks like the HTTP get call to https://www.okx.com/api/v5/account/balance took about 10 seconds during this time.

ntheile commented 2 years ago

Here is another example of a timeout for the same HTTP GET call (10 second timeout) https://www.okx.com/api/v5/account/balance from this trace service.name:galoy-bbw-dealer,name:app.exchangeBase.fetchBalance,error.name:RequestTimeout....but the trace originated from a few different places, causing three separate alerts in pager duty:

  1. name:app.exchangeBase.fetchBalance,error.name:RequestTimeout
  2. name:app.okexExchange.closePosition,error.name:RequestTimeout
  3. name:app.exchangeBase.createMarketOrder

https://ui.honeycomb.io/galoy/datasets/galoy-bbw/result/h9BbZxnEnDN/trace/f7dErqKHSY5?fields[]=c_name&fields[]=c_service.name&span=e05be23255ea7752