search

bisq-network / bisq

A decentralized bitcoin exchange network
https://bisq.network
GNU Affero General Public License v3.0
4.72k stars 1.27k forks source link

Bisq Pricenode crashes on startup if any providers are unavailable for first query #4441

Closed wiz closed 3 years ago

wiz commented 4 years ago

@cd2357 can you please look at this critical issue ASAP?

    __    _                             _                           __
   / /_  (_)________ _      ____  _____(_)_______  ____  ____  ____/ /__
  / __ \/ / ___/ __ `/_____/ __ \/ ___/ / ___/ _ \/ __ \/ __ \/ __  / _ \
 / /_/ / (__  ) /_/ /_____/ /_/ / /  / / /__/  __/ / / / /_/ / /_/ /  __/
/_.___/_/____/\__, /     / .___/_/  /_/\___/\___/_/ /_/\____/\__,_/\___/
                /_/     /_/  (v0.7.2-SNAPSHOT)
bisq/bisq/lib/pricenode-0.7.2-SNAPSHOT.jar started by bisq in /)
#033[0;39m#033[34mAug-27 02:32:43.569 [main] INFO  bisq.price.Main: No active profile set, falling back to default profiles: default
#033[0;39mWARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/bisq/bisq/lib/spring-core-4.3.14.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
#033[34mAug-27 02:32:45.191 [main] INFO  o.s.b.c.e.t.TomcatEmbeddedServletContainer: Tomcat initialized with port(s): 8080 (http)
#033[0;39m#033[34mAug-27 02:32:45.759 [main] INFO  b.p.s.p.Hitbtc: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.766 [main] INFO  b.p.s.p.Coinone: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.771 [main] INFO  b.p.s.p.Bitstamp: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.775 [main] INFO  b.p.s.p.Bitfinex: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.779 [main] INFO  b.p.s.p.CoinGecko: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.789 [main] INFO  b.p.s.p.Quoine: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.791 [main] INFO  b.p.s.p.Luno: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.794 [main] INFO  b.p.s.p.CoinMarketCap: will refresh every PT5M
#033[0;39m#033[34mAug-27 02:32:45.796 [main] INFO  b.p.s.p.Huobi: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.799 [main] INFO  b.p.s.p.Bitflyer: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.801 [main] INFO  b.p.s.p.IndependentReserve: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.804 [main] INFO  b.p.s.p.Paribu: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.806 [main] INFO  b.p.s.p.BitcoinAverage: will refresh every PT1H40M
#033[0;39m#033[34mAug-27 02:32:45.809 [main] INFO  b.p.s.p.Poloniex: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.811 [main] INFO  b.p.s.p.Bitbay: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.813 [main] INFO  b.p.s.p.Binance: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.816 [main] INFO  b.p.s.p.MercadoBitcoin: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.818 [main] INFO  b.p.s.p.Coinpaprika: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.823 [main] INFO  b.p.s.p.Exmo: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.825 [main] INFO  b.p.s.p.Bitpay: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.831 [main] INFO  b.p.s.p.Kraken: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.834 [main] INFO  b.p.s.p.BTCMarkets: will refresh every PT1M
#033[0;39m#033[34mAug-27 02:32:45.852 [main] INFO  b.p.m.p.MempoolFeeRateProvider$First: will refresh every PT2M
#033[0;39m#033[34mAug-27 02:32:45.856 [main] INFO  b.p.m.p.MempoolFeeRateProvider$Second: will refresh every PT2M
#033[0;39m#033[34mAug-27 02:32:51.363 [main] INFO  b.p.s.p.Hitbtc: refresh took 4772 ms.
#033[0;39m#033[34mAug-27 02:32:51.367 [main] INFO  b.p.s.p.Hitbtc: BTC/USD: 11384.16
#033[0;39m#033[34mAug-27 02:32:51.374 [main] INFO  b.p.s.p.Hitbtc: BTC/LTC: 0.0050699
#033[0;39m#033[34mAug-27 02:32:51.670 [main] INFO  b.p.s.p.Coinone: refresh took 295 ms.
#033[0;39m#033[34mAug-27 02:32:52.748 [main] INFO  b.p.s.p.Bitstamp: refresh took 1077 ms.
#033[0;39m#033[34mAug-27 02:32:52.750 [main] INFO  b.p.s.p.Bitstamp: BTC/LTC: 0.00507334
#033[0;39m#033[34mAug-27 02:32:52.751 [main] INFO  b.p.s.p.Bitstamp: BTC/USD: 11388.52
#033[0;39m#033[34mAug-27 02:32:54.302 [main] INFO  b.p.s.p.Bitfinex: refresh took 1550 ms.
#033[0;39m#033[34mAug-27 02:32:54.303 [main] INFO  b.p.s.p.Bitfinex: BTC/USD: 11402.0
#033[0;39m#033[34mAug-27 02:32:54.303 [main] INFO  b.p.s.p.Bitfinex: BTC/LTC: 0.0050748
#033[0;39m#033[34mAug-27 02:32:54.425 [main] INFO  b.p.s.p.CoinGecko: refresh took 122 ms.
#033[0;39m#033[34mAug-27 02:32:54.426 [main] INFO  b.p.s.p.CoinGecko: BTC/LTC: 0.00507406
#033[0;39m#033[34mAug-27 02:32:54.426 [main] INFO  b.p.s.p.CoinGecko: BTC/USD: 11392.673
#033[0;39m#033[34mAug-27 02:32:55.232 [main] INFO  b.p.s.p.Quoine: refresh took 805 ms.
#033[0;39m#033[34mAug-27 02:32:55.233 [main] INFO  b.p.s.p.Quoine: BTC/USD: 11410.33
#033[0;39m#033[34mAug-27 02:33:00.331 [main] INFO  b.p.s.p.Luno: refresh took 5098 ms.
#033[0;39m#033[34mAug-27 02:33:00.331 [main] INFO  b.p.s.p.CoinMarketCap: refresh took 0 ms.
#033[0;39m#033[34mAug-27 02:33:02.690 [main] INFO  b.p.s.p.Huobi: refresh took 2358 ms.
#033[0;39m#033[34mAug-27 02:33:02.691 [main] INFO  b.p.s.p.Huobi: BTC/LTC: 0.005067
#033[0;39m#033[31mAug-27 02:33:03.116 [main] WARN  o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'bitflyer'; nested exception is org.knowm.xchange.exceptions.ExchangeException: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#033[0;39m#033[1;31mAug-27 02:33:03.173 [main] ERROR o.s.b.SpringApplication: Application startup failed org.springframework.context.ApplicationContextException: Failed to start bean 'bitflyer'; nested exception is org.knowm.xchange.exceptions.ExchangeException: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
#011at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50)
#011at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348)
#011at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151)
#011at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114)
#011at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)
#011at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
#011at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
#011at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
#011at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
#011at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
#011at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
#011at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
#011at bisq.price.Main.main(Main.java:36)
Caused by: org.knowm.xchange.exceptions.ExchangeException: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at org.knowm.xchange.bitflyer.service.BitflyerBaseService.handleError(BitflyerBaseService.java:48)
#011at org.knowm.xchange.bitflyer.service.BitflyerMarketDataServiceRaw.getMarkets(BitflyerMarketDataServiceRaw.java:32)
#011at org.knowm.xchange.bitflyer.BitflyerExchange.remoteInit(BitflyerExchange.java:53)
#011at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:107)
#011at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:78)
#011at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:36)
#011at bisq.price.spot.ExchangeRateProvider.doGet(ExchangeRateProvider.java:106)
#011at bisq.price.spot.providers.Bitflyer.doGet(Bitflyer.java:42)
#011at bisq.price.spot.providers.Bitflyer.doGet(Bitflyer.java:31)
#011at bisq.price.PriceProvider.refresh(PriceProvider.java:78)
#011at bisq.price.PriceProvider.start(PriceProvider.java:58)
#011at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175)
#011... 13 common frames omitted
Caused by: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
#011at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
#011at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
#011at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
#011at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124)
#011at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:274)
#011at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:228)
#011at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:138)
#011at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:405)
#011at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65)
#011at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
#011at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3814)
#011at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2896)
#011at si.mazi.rescu.serialization.jackson.JacksonResponseReader.read(JacksonResponseReader.java:53)
#011at si.mazi.rescu.serialization.jackson.JacksonResponseReader.readException(JacksonResponseReader.java:58)
#011at si.mazi.rescu.ResponseReader.read(ResponseReader.java:82)
#011at si.mazi.rescu.RestInvocationHandler.mapInvocationResult(RestInvocationHandler.java:175)
#011at si.mazi.rescu.RestInvocationHandler.receiveAndMap(RestInvocationHandler.java:163)
#011at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java:119)
#011at com.sun.proxy.$Proxy102.getMarkets(Unknown Source)
#011at org.knowm.xchange.bitflyer.service.BitflyerMarketDataServiceRaw.getMarkets(BitflyerMarketDataServiceRaw.java:30)
#011... 23 common frames omitted
#033[0;39morg.springframework.context.ApplicationContextException: Failed to start bean 'bitflyer'; nested exception is org.knowm.xchange.exceptions.ExchangeException: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
#011at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50)
#011at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348)
#011at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151)
#011at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114)
#011at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)
#011at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
#011at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
#011at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
#011at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
#011at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
#011at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
#011at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
#011at bisq.price.Main.main(Main.java:36)
Caused by: org.knowm.xchange.exceptions.ExchangeException: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at org.knowm.xchange.bitflyer.service.BitflyerBaseService.handleError(BitflyerBaseService.java:48)
#011at org.knowm.xchange.bitflyer.service.BitflyerMarketDataServiceRaw.getMarkets(BitflyerMarketDataServiceRaw.java:32)
#011at org.knowm.xchange.bitflyer.BitflyerExchange.remoteInit(BitflyerExchange.java:53)
#011at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:107)
#011at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:78)
#011at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:36)
#011at bisq.price.spot.ExchangeRateProvider.doGet(ExchangeRateProvider.java:106)
#011at bisq.price.spot.providers.Bitflyer.doGet(Bitflyer.java:42)
#011at bisq.price.spot.providers.Bitflyer.doGet(Bitflyer.java:31)
#011at bisq.price.PriceProvider.refresh(PriceProvider.java:78)
#011at bisq.price.PriceProvider.start(PriceProvider.java:58)
#011at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175)
#011... 13 common frames omitted
Caused by: org.knowm.xchange.bitflyer.dto.BitflyerException: Under maintenance
#011at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
#011at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
#011at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
#011at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
#011at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124)
#011at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:274)
#011at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:228)
#011at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:138)
#011at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:405)
#011at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65)
#011at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
#011at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3814)
#011at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2896)
#011at si.mazi.rescu.serialization.jackson.JacksonResponseReader.read(JacksonResponseReader.java:53)
#011at si.mazi.rescu.serialization.jackson.JacksonResponseReader.readException(JacksonResponseReader.java:58)
#011at si.mazi.rescu.ResponseReader.read(ResponseReader.java:82)
#011at si.mazi.rescu.RestInvocationHandler.mapInvocationResult(RestInvocationHandler.java:175)
#011at si.mazi.rescu.RestInvocationHandler.receiveAndMap(RestInvocationHandler.java:163)
#011at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java:119)
#011at com.sun.proxy.$Proxy102.getMarkets(Unknown Source)
#011at org.knowm.xchange.bitflyer.service.BitflyerMarketDataServiceRaw.getMarkets(BitflyerMarketDataServiceRaw.java:30)
#011... 23 common frames omitted
cd2357 commented 4 years ago

Pricenode crashes on startup if any providers are unavailable for first query

That's part of the general pricenode architecture. It is not related to my recent change.

It would also "crash" (i.e. fail to start) if a fee rate provider like mempool.space would return any issues. In fact, even before the recent change, it would have failed to start if the first BA or Poloniex queries would have failed.

So its part of the general architecture of how @Components (providers) are initialized. If it fails to initialize, it retries a couple times, but if it cannot get them to properly initialize, it assumes smth is wrong and aborts.

The behavior does make sense: if the pricenode starts, but cannot get a specific price, what would it return to the Bisq nodes? A price of 0? It also has no cached prices from previous queries, cause this is on startup.

So from what I can tell, changing this behavior would:

cd2357 commented 4 years ago

A simple alternative / workaround would be to add the ability to ignore or skip some providers on startup, like

./bisq-pricenode --skipProviders=Bitflyer

But that is also not trivial:

So this could be a simpler alternative, but also not that straightforward.

wiz commented 4 years ago

That's part of the general pricenode architecture. It is not related to my recent change.

Yes, I am not blaming you, I'm just asking you to fix this since you are familiar with the pricenode code :)

A simple alternative / workaround would be to add the ability to ignore or skip some providers on startup, like

No, the pricenode always needs to startup 100% of the time. It's insane that it can fail to start if not all providers are responding.

The behavior does make sense: if the pricenode starts, but cannot get a specific price, what would it return to the Bisq nodes? A price of 0? It also has no cached prices from previous queries, cause this is on startup.

Can't you simply handle the exception and set that provider's weight to zero until the provider starts returning valid data again?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

devinbileck commented 3 years ago

I restarted my pricenode and it kept crashing due to issues with Binance's API. Apparently it was undergoing maintenance. See https://www.binance.com/en/support/announcement/1344e0dd30e843a8801aa58e23d6d82e. binance crash.txt

In order to get it running, I removed the binance provider but then encountered different errors with huobi, though it didnt crash it. huobi error.txt