Improve retry logic of getting subscription details from internal REST API when returning 404

[dushaniw]

Describe your problem(s)

Due to possible slowness in the server startup, in some cases following log is printed and APIs cannot be seen in devportal and publisher.

ERROR {org.wso2.carbon.apimgt.throttle.policy.deployer.utils.PolicyUtil} - Error in retrieving throttle policies org.wso2.carbon.apimgt.throttle.policy.deployer.exception.ThrottlePolicyDeployerException: Event-Hub status code is : 404
at org.wso2.carbon.apimgt.throttle.policy.deployer.PolicyRetriever.getPolicies(PolicyRetriever.java:203)
at org.wso2.carbon.apimgt.throttle.policy.deployer.PolicyRetriever.getAllSubscriptionPolicies(PolicyRetriever.java:87)
at org.wso2.carbon.apimgt.throttle.policy.deployer.utils.PolicyUtil.deployAllPolicies(PolicyUtil.java:159)
at org.wso2.carbon.apimgt.throttle.policy.deployer.utils.ThrottlePolicyStartupListener.run(ThrottlePolicyStartupListener.java:69)
at java.lang.Thread.run(Thread.java:750)
TID: [-1] [] [2022-04-05 11:20:20,305] ERROR {org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl} - Could not retrieve subscriptions for tenantDomain : carbon.super
TID: [-1] [] [2022-04-05 11:20:20,475] ERROR {org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataStoreImpl} - Exception while loading Subscriptions org.wso2.carbon.apimgt.keymgt.model.exception.DataLoadingException: Error while retrieving subscription from /subscriptions
TID: [-1] [] [2022-04-05 11:20:20,512] ERROR {org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl} - Could not retrieve subscriptions for tenantDomain : carbon.super

Describe your solution

Improve the retry logic of http client call when the internal APIs return a 404 error.

[dushaniw]

Wont fix in master as it will break choreo connect.