Metaspace OOM after repeated hot deployment of Spring Boot war file in Tomcat

[sywong70g]

Environment:

a. OS : Ubuntu 22.04.3 b. JVM : Oracle Java 11.0.18 and Oracle java 17.0.17 (tested both) c. Tomcat : 9.0.75 d. SpringBoot : 2.7.5 e. Oracle : version 19c f. Oracle jdbc : com.oracle.database.jdbc:ojdbc11:23.2.0.0

Source code:

Policy.java:

Getter
@Setter
@Entity
@Table(name = "policy_m")
public class Policy {

   @Column(name = "policy_agreement_id")
   private Long policyAgreementId;
   @Id
   @Column(name = "policy_num")
   private String policyNum;

}

PolicyRepository.java:

@Repository
public interface PolicyRepository extends JpaRepository<Policy, String> {

   @Query(value = "select * from policy_m where policy_num = :policyNum", nativeQuery = true)
   Policy getPolicyByPolicyNum(@Param("policyNum") String policyNum);

}

PolicyServiceImpl.java:

@Service
public class PolicyServiceImpl implements PolicyService {

   @Autowired
   private PolicyRepository policyRepository;

   public Policy getPolicyByPolicyNum(String policyNum) {
       return policyRepository.getPolicyByPolicyNum(policyNum);
   }
}

DemoController.java:

@RestController
public class DemoController {

   @Autowired
   private PolicyService policyService;

   @GetMapping("/{policyNum}")
   public Policy database(@PathVariable("policyNum") String policyNum) {
       return policyService.getPolicyByPolicyNum(policyNum);
   }
}

Steps to reproduce:

1. prepare the war file
2. deploy to the tomcat through the admin console
3. undeploy the application
4. repeat step 2 - 3 until it throws MetaSpace OOM

The VisualVM shows the chart:

At the plateau at the right hand side, the tomcat actually already hanged.

[wilkinsona]

Tomcat's pretty good at identifying possible memory leaks and logging about them. Is there anything in Tomcat's logs when you undeploy the app? If there isn't (and perhaps even if there is) I think we'll need a sample that reproduces the problem, ideally one that doesn't require an Oracle DB.

[sywong70g]

I just tested with a helloWorld application without DB connection and actually there is no such issue. I am wondering whether it is about the connection pool. Also I am testing on MySQL with JPA and see if the same issue occurs. If yes, I will send you the sample.

[quaff]

It's very likely bug of jdbc driver, you can switch to another database and test.

Also you can try to extend SpringBootServletInitializer and override deregisterJdbcDrivers

    @Override
    protected void deregisterJdbcDrivers(ServletContext servletContext) {
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        try {
            String className = "com.mysql.cj.jdbc.AbandonedConnectionCleanupThread";
            String methodName = "checkedShutdown";
            if (ClassUtils.isPresent(className, cl)) {
                ClassUtils.forName(className, cl).getMethod(methodName).invoke(null);
            }
        }
        catch (Throwable ex) {
            ex.printStackTrace();
        }
        try {
            String className = "com.mysql.jdbc.AbandonedConnectionCleanupThread";
            String methodName = "checkedShutdown";
            if (ClassUtils.isPresent(className, cl)) {
                ClassUtils.forName(className, cl).getMethod(methodName).invoke(null);
            }
        }
        catch (Throwable ex) {
            ex.printStackTrace();
        }
        super.deregisterJdbcDrivers(servletContext);
        cancelTimers();
        cleanupThreadLocals();
    }

    protected void cancelTimers() {
        try {
            for (Thread thread : Thread.getAllStackTraces().keySet()) {
                if (thread.getClass().getSimpleName().equals("TimerThread")) {
                    cancelTimer(thread);
                }
            }
        }
        catch (Throwable ex) {
            ex.printStackTrace();
        }
    }

    private void cancelTimer(Thread thread) throws Exception {
        Object queue = ReflectionUtils.getFieldValue(thread, "queue");
        Method m = queue.getClass().getDeclaredMethod("isEmpty");
        m.setAccessible(true);
        if ((boolean) m.invoke(queue)) {
            // Timer::cancel
            synchronized (queue) {
                ReflectionUtils.setFieldValue(thread, "newTasksMayBeScheduled", false);
                m = queue.getClass().getDeclaredMethod("clear");
                m.setAccessible(true);
                m.invoke(queue);
                queue.notify();
            }
        }
    }

    protected void cleanupThreadLocals() {
        try {
            for (Thread thread : Thread.getAllStackTraces().keySet()) {
                cleanupThreadLocals(thread);
            }
        }
        catch (Throwable ex) {
            ex.printStackTrace();
        }
    }

    private void cleanupThreadLocals(Thread thread) throws Exception {
        if ("JettyShutdownThread".equals(thread.getName())) {
            return; // see https://github.com/eclipse/jetty.project/issues/5782
        }
        for (String name : "threadLocals,inheritableThreadLocals".split(",")) {
            Field f = Thread.class.getDeclaredField(name);
            f.setAccessible(true);
            f.set(thread, null);
        }
    }

[sywong70g]

@quaff you may be right, I use the same source code but connect to MySQL instead of Oracle, there is no such issue.

Thank you for your suggestion, we are testing your code and see if we can solve the issue.

[philwebb]

Thanks for your help @quaff. I'll close this one for now, but if it turns out to be something in Spring Boot rather than the Oracle driver we can reopen it.