[Bitfinex] Duplicate key DOGE/USD

[piechos92]

When invoking applySpecification() method on exchange I get this exception:

Caused by: java.lang.IllegalStateException: Duplicate key DOGE/USD (attempted merging values 0.086684 and 0.30382) at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:133) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:442) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.knowm.xchange.bitfinex.BitfinexExchange.remoteInit(BitfinexExchange.java:86) at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:113)

I debugged the issue and it seems like 2 CurrencyPairs: XDC/USD and DOGE/USD are treated as equal when putting them in the map by stream collector in this piece of code (I suspect that because these currency pairs have same values as in stacktrace - 0.086684 and 0.30382)

Map<CurrencyPair, BigDecimal> lastPrices =
          Arrays.stream(dataService.getBitfinexTickers(null))
              .map(BitfinexAdapters::adaptTicker)
              .peek(t -> System.out.println(String.format(">>>  %s >>  %s",t.getCurrencyPair(), t.getLast())))
              .collect(Collectors.toMap(t -> t.getCurrencyPair(), t -> t.getLast()))

I see that CurrencyPair has some custom equals() method so probably the issue lies there

[earce]

There was recently a ticket addressing issues with doge's symbology, can you attempt to run your code on snapshot build and see if it solves your issue https://github.com/knowm/XChange/pull/4023

[piechos92]

I run this with development branch and now I get exception few lines further

io.taxtoken.core.exception.ExchangeApiIntegrationException: null at io.taxtoken.core.dataintegration.api.exchange.XChangeAbstractIntegrationService.integrateTransactions(XChangeAbstractIntegrationService.java:95) at io.taxtoken.core.dataintegration.api.exchange.IntegrationExchangeApiService.performIntegration(IntegrationExchangeApiService.java:71) at io.taxtoken.core.dataintegration.AbstractIntegrationService.integrate(AbstractIntegrationService.java:137) at io.taxtoken.integrator.IntegrationTaskController.integrateTransactions(IntegrationTaskController.java:152) at io.taxtoken.integrator.IntegrationTaskController.performTask(IntegrationTaskController.java:83) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908) at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:679) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.knowm.xchange.exceptions.ExchangeException: si.mazi.rescu.HttpStatusIOException: HTTP status code was not OK: 500 at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:115) at io.taxtoken.core.dataintegration.api.exchange.XChangeAbstractIntegrationService.createExchange(XChangeAbstractIntegrationService.java:115) at io.taxtoken.core.dataintegration.api.exchange.XChangeAbstractIntegrationService.integrateTransactions(XChangeAbstractIntegrationService.java:85) ... 74 common frames omitted Caused by: si.mazi.rescu.HttpStatusIOException: HTTP status code was not OK: 500 at si.mazi.rescu.ResponseReader.read(ResponseReader.java:104) at si.mazi.rescu.RestInvocationHandler.mapInvocationResult(RestInvocationHandler.java:169) at si.mazi.rescu.RestInvocationHandler.receiveAndMap(RestInvocationHandler.java:157) at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java:113) at com.sun.proxy.$Proxy212.accountFees(Unknown Source) at org.knowm.xchange.bitfinex.service.BitfinexAccountServiceRaw.lambda$getAccountFees$6(BitfinexAccountServiceRaw.java:211) at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:306) at org.knowm.xchange.client.ResilienceUtils$CallableApi.lambda$wrapCallable$0(ResilienceUtils.java:40) at io.github.resilience4j.ratelimiter.RateLimiter.lambda$decorateCheckedSupplier$9076412b$1(RateLimiter.java:215) at io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247) at io.vavr.Function0.get(Function0.java:149) at io.github.resilience4j.ratelimiter.RateLimiter.lambda$decorateCallable$6(RateLimiter.java:442) at org.knowm.xchange.client.ResilienceUtils$CallableApi.lambda$wrapCallable$0(ResilienceUtils.java:40) at org.knowm.xchange.client.ResilienceUtils$DecorateCallableApi.call(ResilienceUtils.java:86) at org.knowm.xchange.bitfinex.service.BitfinexAccountServiceRaw.getAccountFees(BitfinexAccountServiceRaw.java:220) at org.knowm.xchange.bitfinex.BitfinexExchange.remoteInit(BitfinexExchange.java:95) at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:113) ... 76 common frames omitted

[earce]

This is something else si.mazi.rescu.HttpStatusIOException: HTTP status code was not OK: 500

[piechos92]

Yes. Was this issue known already?

[earce]

I am not aware of this issue. Not sure if anyone else is, you can try the discord group.

[walec51]

the exchange returned 500 error code which indicates and internal error in the exchange it self

it may caused by you account specific data

please set the logging level so we can see the full exchanges response and paste it here

https://github.com/mmazi/rescu#logging