Datafeed retry loop fails after receiving KeyManager exception

[ystxn]

Bug Report

Steps to Reproduce:

Keep bot running during a pod maintenance window where components like SBE and Agent are taken offline.

Expected Result:

• The bot should continue retrying the datafeed loop throughout the duration of the outage
• When all components are brought online and operational, the next retry should re-establish connections and bring the bot back online

Actual Result:

• The bot retries continuously when it receives error codes 502 and 503
• When the bot receives the following exception, it stops the retry loop
• When the pod returns to operational status, the bot is still frozen in the above state and does not re-establish connections

Environment:

• BDK Spring Starter 2.8.0

Exception:

2022-09-14 23:54:34.291 ERROR 1 --- [onyBdk_Datafeed] c.s.bdk.core.retry.RetryWithRecovery     : An unknown error occurred while trying to connect to https://corporate.symphony.com:443/relay. Please check below for more information: Not allowed to make this call when KeyManager is not registered in the pod
2022-09-14 23:54:34.514 ERROR 1 --- [onyBdk_Datafeed] c.s.b.c.s.d.impl.AbstractDatafeedLoop    : An unknown error occurred while trying to connect to https://corporate.symphony.com:443/agent. Please check below for more information:
{}
com.symphony.bdk.http.api.ApiRuntimeException: com.symphony.bdk.http.api.ApiException: Not allowed to make this call when KeyManager is not registered in the pod
        at com.symphony.bdk.core.auth.impl.AuthenticationRetry.executeAndRetry(AuthenticationRetry.java:90) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.AbstractBotAuthenticator.retrieveKeyManagerToken(AbstractBotAuthenticator.java:48) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.BotAuthenticatorRsaImpl.retrieveKeyManagerToken(BotAuthenticatorRsaImpl.java:66) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.AuthSessionImpl.refreshAllTokens(AuthSessionImpl.java:105) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.AuthSessionImpl.refresh(AuthSessionImpl.java:87) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.service.datafeed.impl.AbstractDatafeedLoop.refresh(AbstractDatafeedLoop.java:155) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.retry.RecoveryStrategy.runRecovery(RecoveryStrategy.java:48) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.retry.RetryWithRecovery.handleRecovery(RetryWithRecovery.java:159) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.retry.RetryWithRecovery.executeOnce(RetryWithRecovery.java:112) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$3f69f149$1(Retry.java:137) ~[resilience4j-retry-1.7.1.jar!/:1.7.1]
        at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:419) ~[resilience4j-retry-1.7.1.jar!/:1.7.1]
        at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.execute(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.service.datafeed.impl.DatafeedLoopV2.runLoop(DatafeedLoopV2.java:110) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.service.datafeed.impl.AbstractDatafeedLoop.start(AbstractDatafeedLoop.java:87) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.service.datafeed.impl.AbstractAckIdEventLoop.start(AbstractAckIdEventLoop.java:19) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.spring.service.LoopAsyncLauncherService.uncheckedStart(LoopAsyncLauncherService.java:87) ~[symphony-bdk-core-spring-boot-starter-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.http.api.tracing.MDCUtils$MdcRunnable.run(MDCUtils.java:59) ~[symphony-bdk-http-api-2.8.0.jar!/:2.8.0]
        at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: com.symphony.bdk.http.api.ApiException: Not allowed to make this call when KeyManager is not registered in the pod
        at com.symphony.bdk.http.jersey2.ApiClientJersey2.invokeAPI(ApiClientJersey2.java:192) ~[symphony-bdk-http-jersey2-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.gen.api.AuthenticationApi.pubkeyAuthenticatePostWithHttpInfo(AuthenticationApi.java:496) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]        at com.symphony.bdk.gen.api.AuthenticationApi.pubkeyAuthenticatePost(AuthenticationApi.java:443) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.BotAuthenticatorRsaImpl.doRetrieveToken(BotAuthenticatorRsaImpl.java:75) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.AbstractBotAuthenticator.lambda$retrieveKeyManagerToken$0(AbstractBotAuthenticator.java:49) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.retry.RetryWithRecovery.executeOnce(RetryWithRecovery.java:105) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$3f69f149$1(Retry.java:137) ~[resilience4j-retry-1.7.1.jar!/:1.7.1]
        at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:419) ~[resilience4j-retry-1.7.1.jar!/:1.7.1]
        at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.execute(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        at com.symphony.bdk.core.auth.impl.AuthenticationRetry.executeAndRetry(AuthenticationRetry.java:85) ~[symphony-bdk-core-2.8.0.jar!/:2.8.0]
        ... 17 common frames omitted