search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.75k stars 1.81k forks source link

[BUG] Uncaught exception in main thread doesn't halt JVM #15241

Open Bukhtawar opened 2 months ago

Bukhtawar commented 2 months ago

Describe the bug

[2024-08-13T19:53:41,233][ERROR][o.o.b.Bootstrap          ] [a28ce3bd16681e0092591169b3fd0b0a] Exception
java.lang.IllegalStateException: Unable to restore remote index metadata
        at org.opensearch.index.recovery.RemoteStoreRestoreService.restore(RemoteStoreRestoreService.java:158)
        at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:178)
        at org.opensearch.node.Node.start(Node.java:1546)
        at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
[2024-08-14T00:04:18,880][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [ee3fe29191a528a0f473bb2fa9b77f12] uncaught exception in thread [main]
java.lang.IllegalStateException: Unable to restore remote index metadata

The above stack trace was noted in a non-fatal exception(not an instance of Error) on main thread which didn't halt the JVM keeping the process in a stuck state. It would have been preferable to shut the running process to allow monitoring or nanny scripts to restart the process.

https://github.com/opensearch-project/OpenSearch/blob/411b279e8d8c25ff26a961faacf0ee1fd9d0e612/server/src/main/java/org/opensearch/bootstrap/OpenSearchUncaughtExceptionHandler.java#L53-L77

Related component

Storage:Remote

To Reproduce

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior

JVM should halt on uncaught exception on main thread

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

Additional context Add any other context about the problem here.

andrross commented 2 months ago

I wonder if we should be catching arbitrary exceptions here and returning an error exit code?

https://github.com/opensearch-project/OpenSearch/blob/c0852f84f0647f78a45e71396d6370a39ba4922f/libs/cli/src/main/java/org/opensearch/cli/Command.java#L100-L116