Errors on Hot Reloading (Java)

[rshmhrj]

Describe the bug

This config works fine normally, but when we are working on a dev laptop, with devtools running, saving changes within the codebase causes a hot reload. The UnleashClientConfig reloads and there is still an unleash bean running in the ApplicationContext. The ConditionalOnMissingBean doesn't trigger on Hot Reload, so new DefaultUnleash(unleashConfig) runs and spits out some errors:

2024-05-12 02:41:43,889 ERROR [ ] i.g.u.UnleashScheduledExecutorImpl: Unleash background task crashed java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2cbea797[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4e329255[Wrapped task = io.getunleash.metric.UnleashMetricServiceImpl$$Lambda$2187/0x000000f801aef1d0@31ae30c6]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@4d457db9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:632)
at io.getunleash.util.UnleashScheduledExecutorImpl.setInterval(UnleashScheduledExecutorImpl.java:43)
at io.getunleash.metric.UnleashMetricServiceImpl.<init>(UnleashMetricServiceImpl.java:32)   
at io.getunleash.metric.UnleashMetricServiceImpl.<init>(UnleashMetricServiceImpl.java:20)
       at io.getunleash.DefaultUnleash.<init>(DefaultUnleash.java:67)
       at io.getunleash.DefaultUnleash.<init>(DefaultUnleash.java:54)
       at com.example.config.UnleashClientConfig$UnleashInstanceCreation.unleash(UnleashClientConfig.java:115)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
      at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:493)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
           at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
          at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
      at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
          at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:941)
      at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)
          at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:733)
  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:435)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:311)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1305)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1294)
        at com.example.Application.main(Application.java:11)
              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
      at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) 

You already have 2 clients for Unleash Configuration [apiKey:[unleash_poc_test:development.71e80e487cc0978bf98c03d0898f34479a933188665bcc07fa19f9c7] appName:[development] instanceId:[d429963b-f006-48c0-9a3b-058b8cf411ac]] running. Please double check your code where you are instantiating the Unleash SDK

Every time we save, it throws the errors.

package com.example.config;

import io.getunleash.DefaultUnleash;
import io.getunleash.Unleash;
import io.getunleash.UnleashContextProvider;
import io.getunleash.repository.ToggleBootstrapFileProvider;
import io.getunleash.util.UnleashConfig;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@AutoConfiguration
@Configuration
public class UnleashClientConfig {
    @Value("${unleash.backupFile}") String backupFile;
    @Value("${unleash.fetchIntervalSeconds}") Long fetchInterval;

    @Bean
    @ConditionalOnMissingBean(UnleashConfig.class)
    public UnleashConfig unleashConfig(
            @Value("${unleash.appName:development}") String appName,
            @Value("${unleash.instanceId:d429963b-f006-48c0-9a3b-058b8cf411ac}") String instanceId,
            @Value("${unleash.apiUrl:https://app.unleash-hosted.com/demo/api}") String apiUrl,
            @Value("${unleash.apiKey:unleash_poc_test:development.71e80e487cc0978bf98c03d0898f34479a933188665bcc07fa19f9c7}") String apiKey,
            UnleashContextProvider unleashContextProvider) {
        log.debug("Unleash config initialized with backupFile: {}, fetchInterval: {}", backupFile, fetchInterval);
        log.debug("Unleash config initialized with appName: {}, instanceId: {}, apiUrl: {}", appName, instanceId, apiUrl);
        return UnleashConfig.builder()
                .toggleBootstrapProvider(new ToggleBootstrapFileProvider(backupFile))
                .appName(appName)
                .instanceId(instanceId)
                .unleashAPI(apiUrl)
                .apiKey(apiKey)
                .unleashContextProvider(unleashContextProvider)
                .backupFile(backupFile)
                .fetchTogglesInterval(fetchInterval)
                .build();
    }

    @Bean
    @ConditionalOnMissingBean(Unleash.class)
    public Unleash unleash(UnleashConfig unleashConfig) {
        return new DefaultUnleash(unleashConfig);
    }
}

I tried fixing by checking to see if the bean was already running and kept facing circular dependency errors. Tried with the normal singleton pattern, @Scope("singleton"), trying to find the bean from the ApplicationContext and they all kept failing for circular dependencies.

With the below code change, the server starts, but as soon as I hit an endpoint and the bean is initialized, it fails with BeanCreationException:

2024-05-12 03:04:29,251 WARN  [] o.s.w.s.h.AbstractHandlerExceptionResolver: 
Resolved [org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'unleash' defined in class path resource [com/example/config/UnleashClientConfig$UnleashInstanceCreation.class]: 
Failed to instantiate [io.getunleash.Unleash]: 
Factory method 'unleash' threw exception with message: 
Error creating bean with name 'unleash': 
Requested bean is currently in creation: 
Is there an unresolvable circular reference?] 

package com.example.config;

import org.springframework.beans.factory.BeanCurrentlyInCreationException;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Lazy;

import io.getunleash.DefaultUnleash;
import io.getunleash.Unleash;
import io.getunleash.UnleashContextProvider;
import io.getunleash.repository.ToggleBootstrapFileProvider;
import io.getunleash.util.UnleashConfig;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@AutoConfiguration
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public class UnleashClientConfig {

    @Configuration(proxyBeanMethods=false)
    public static class UnleashConfiguration {
        @Value("${unleash.backupFile:backup/unleash.bkp}") String backupFile;
        @Value("${unleash.fetchIntervalSeconds:300}") Long fetchInterval;

        @Bean
        @ConditionalOnMissingBean(UnleashConfig.class)
        public UnleashConfig unleashConfig(
                @Value("${unleash.appName:development}") String appName,
                @Value("${unleash.instanceId:d429963b-f006-48c0-9a3b-058b8cf411ac}") String instanceId,
                @Value("${unleash.apiUrl:https://app.unleash-hosted.com/demo/api}") String apiUrl,
                @Value("${unleash.apiKey:unleash_poc_test:development.71e80e487cc0978bf98c03d0898f34479a933188665bcc07fa19f9c7}") String apiKey,
                UnleashContextProvider unleashContextProvider) {
            log.debug("Unleash config initialized with backupFile: {}, fetchInterval: {}", backupFile, fetchInterval);
            log.debug("Unleash config initialized with appName: {}, instanceId: {}, apiUrl: {}", appName, instanceId, apiUrl);
            return UnleashConfig.builder()
                    .toggleBootstrapProvider(new ToggleBootstrapFileProvider(backupFile))
                    .appName(appName)
                    .instanceId(instanceId)
                    .unleashAPI(apiUrl)
                    .apiKey(apiKey)
                    .unleashContextProvider(unleashContextProvider)
                    .backupFile(backupFile)
                    .fetchTogglesInterval(fetchInterval)
                    .synchronousFetchOnInitialisation(false)
                    .build();
        }
    }

    @Configuration(proxyBeanMethods=false)
    @RequiredArgsConstructor(onConstructor = @__(@Autowired))
    public static class UnleashInstanceCreation {
        private final ObjectProvider<Unleash> unleashProvider;
        private final UnleashConfig unleashConfig;

        @Bean
        @Lazy
        @ConditionalOnMissingBean(Unleash.class)
        public Unleash unleash() {
            Unleash unleash = unleashProvider.getIfAvailable();
            if (unleash == null) unleash = new DefaultUnleash(unleashConfig);
            return unleash;
        }
    }

}

Steps to reproduce the bug

No response

Expected behavior

I saw in the docs that there is a shutdown() method which could probably be used, but I'm not sure where or how to set that up. Is there any known way of dealing with the bean recreation during hot reloading?

Logs, error output, etc.

No response

Screenshots

No response

Additional context

No response

Unleash version

5.9.6

Subscription type

Open source

Hosting type

Self-hosted

SDK information (language and version)

Java

[chriswk]

Hi, we don't consider this a bug. Better support for hot reload is a feature request and we don't currently have capacity to do this. We do have capacity to look at a PR though if you'd like to have a stab at fixing it yourself.