Can't create CRaC checkpoint, fails with the error message "Restarting Spring-managed lifecycle beans after JVM restore"

[magnus-larsson]

spring-data-mongodb seems to prevent creating CRaC checkpoints.

Pull request 35 in the spring-lifecycle-smoke-tests - project can be used to reproduce the problem. The pull request contains the comment "Currently failing due to open connections held by the MongoDB driver.".

sdk use java 17.0.9.crac-zulu 
git clone https://github.com/spring-projects/spring-lifecycle-smoke-tests.git
cd spring-lifecycle-smoke-tests/
git fetch origin pull/35/head:PR35
git switch PR35
./gradlew data:data-mongodb:checkpointRestoreAppTest

The CRaC test fails immediately after the checkpointing is started with:

 2024-05-27T10:23:46.519+02:00  INFO 2406006 --- [           main] jdk.crac                                 : Starting checkpoint
 Stopping DataReader: was STARTED
 count 001: Author{id='id-2', name='Brent Weeks'}
 2024-05-27T10:23:46.566+02:00  INFO 2406006 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Restarting Spring-managed lifecycle beans after JVM restore
 Starting DataReader: was STOPPED
 2024-05-27T10:23:46.567+02:00  INFO 2406006 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Spring-managed lifecycle restart completed (restored JVM running for -1 ms)
 2024-05-27T10:23:46.579+02:00 ERROR 2406006 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.context.ApplicationContextException: Failed to take CRaC checkpoint on refresh
    at org.springframework.cr.listener.CheckpointListener$CracDelegate.checkpointRestore(CheckpointListener.java:59) ~[cr-listener.jar!/:na]

To compare with, the spring-data-jpa - CRaC test run fine with the commands:

git switch main
./gradlew data:data-jpa:checkpointRestoreAppTest

[christophstrobl]

Thank you for getting in touch. Right, the MongoDB driver opens connections to the database on create. At the moment there's no out of the box solution that allows to capture an in flight snapshot with the driver being initialized. You may want to ping the MongoDB driver team on JAVA-5034 asking for enhanced lifecycle support. Meanwhile, you can give the spring-cloud @RefreshScope a try.

[magnus-larsson]

Thanks for the update, I'll try out spring-cloud @RefreshScope!

[magnus-larsson]

Hello again!

I noticed that this issue was closed. Does that mean you don't plan to support CRaC?

Anyhow, regarding the MongoDB driver ticket JAVA-5034, they seem to have been waiting on a response from the reporter before closing the ticket due to inactivity:

Hi cstrobl@vmware.com! JAVA-5034 is awaiting your response.

Can the reporter give feedback to the ticket JAVA-5034 to move the discussions forward?

Based on your recommendations, I have performed some tests with spring-cloud-refresh. The problem can be partly resolved by closing the Mongo Client before the checkpoint using CRaC's callback method beforeCheckpoint. To restart the Mongo client after a restore from a CRaC checkpoint, spring-cloud-refresh can be configured to refresh com.mongodb.client.MongoClient.

Unfortunately, the 'mongoClient' does not pick up new configuration parameters for the MongoDB connection provided when the app is restored from a checkpoint.

I have published a project on GitHub that can be used to verify my partial solution; see https://github.com/magnus-larsson/ml-spring-data-mongodb-crac-error-demo. The README.md file contains instructions.

I don't understand what other Spring Beans I need to refresh to make the MongoClient pick-up configuration provided at restore from a checkpoint, e.g., the hostname of the MongoDB database.

Can you help out?

[christophstrobl]

Thank you for circling back to this issue. Regarding @RefreshScope and configuration property changes please get in touch with the spring-cloud team. Regarding the underlying issue, please feel free to reopen the existing JAVA-5034 ticket or create a new one outlining the requirements/expectations towards the MongoClient. Thank you!

[mp911de]

IIRC, MongoProperties would have to be reloaded. CRaC snapshots property values and therefore, any previously stored properties hold the values at the time of the snapshot.

What could work is configuring the driver from Environment (assuming the runtime has picked up the new values from env variables).

[magnus-larsson]

Thanks for your help!

I have been trying to refresh MongoProperties and use environment variables. The configuration is updated, as expected, but the MongoClient does not pick up the updated configuration when the application is resumed from the CRaC checkpoint (as pointed out in JAVA-5034).

I have added a comment to JAVA-5034. But I can't reopen the ticket, so I'm unsure if my comment will get any attention.

@christophstrobl: As a reporter, can you reopen the issue referring to my comment?

[mp911de]

I cannot update the ticket status of MongoDB tickets. Please also note that in a Spring Boot arrangement, Spring Boot handles the client's configuration, and all code to initialize config properties is Spring Boot code.