search

spring-projects / spring-ai

An Application Framework for AI Engineering
https://docs.spring.io/spring-ai/reference/1.0-SNAPSHOT/index.html
Apache License 2.0
2.89k stars 721 forks source link

Tried to specify the thread pool when creating an OrtEnvironment, but one already exists #1020

Open brucewkz opened 2 months ago

brucewkz commented 2 months ago

I built an embedding serving web app using Spring AI + OnnxRuntime, the app is started normally，but when I request the http://127.0.0.1/embedding I get the following error：

(base) ubuntu@ubuntu:~/onnxruntimeDemo$ java -jar onnxruntimeDemo-1.0-SNAPSHOT.jar

. _ /\ / ' _ () _ \ \ \ ( ( )_ | ' | '| | ' / ` | \ \ \ \/ )| |)| | | | | || (| | ) ) ) ) ' || .|| ||| |_, | / / / / =========||==============|/=////

:: Spring Boot :: (v3.3.1)

2024-07-09T03:56:16.098Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : Starting OnnxRuntimeApplication using Java 17.0.9 with PID 1003336 (/home/ubuntu/onnxruntimeDemo/onnxruntimeDemo-1.0-SNAPSHOT.jar started by ubuntu in /home/ubuntu/onnxruntimeDemo) 2024-07-09T03:56:16.104Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : No active profile set, falling back to 1 default profile: "default" 2024-07-09T03:56:16.819Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port 8080 (http) 2024-07-09T03:56:16.830Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.StandardService : Starting service [Tomcat] 2024-07-09T03:56:16.830Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.25] 2024-07-09T03:56:16.859Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext 2024-07-09T03:56:16.861Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 727 ms 2024-07-09T03:56:21.022Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] a.d.u.Platform : Found matching platform from: jar:file:/home/ubuntu/onnxruntimeDemo/lib/tokenizers-0.26.0.jar!/native/lib/tokenizers.properties 2024-07-09T03:56:22.829Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model input names: input_ids, attention_mask 2024-07-09T03:56:22.831Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model output names: token_embeddings, sentence_embedding 2024-07-09T03:56:24.361Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model input names: input_ids, attention_mask 2024-07-09T03:56:24.361Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model output names: token_embeddings, sentence_embedding 2024-07-09T03:56:24.656Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat started on port 8080 (http) with context path '/' 2024-07-09T03:56:24.666Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : Started OnnxRuntimeApplication in 8.922 seconds (process running for 9.46) 2024-07-09T03:56:45.723Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.a.c.c.C.[.[.[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2024-07-09T03:56:45.724Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.s.w.s.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2024-07-09T03:56:45.727Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.s.w.s.DispatcherServlet : Completed initialization in 3 ms 2024-07-09T03:56:45.865Z ERROR 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.] with root cause

java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists. at ai.onnxruntime.OrtEnvironment.getEnvironment(OrtEnvironment.java:136) ~[onnxruntime_gpu-1.14.0.jar:1.14.0] at ai.djl.onnxruntime.engine.OrtEngine.(OrtEngine.java:56) ~[onnxruntime-engine-0.26.0.jar:?] at ai.djl.onnxruntime.engine.OrtEngine.newInstance(OrtEngine.java:64) ~[onnxruntime-engine-0.26.0.jar:?] at ai.djl.onnxruntime.engine.OrtEngineProvider.getEngine(OrtEngineProvider.java:43) ~[onnxruntime-engine-0.26.0.jar:?] at ai.djl.engine.Engine.getEngine(Engine.java:216) ~[api-0.26.0.jar:?] at ai.djl.engine.Engine.getInstance(Engine.java:149) ~[api-0.26.0.jar:?] at ai.djl.ndarray.NDManager.newBaseManager(NDManager.java:120) ~[api-0.26.0.jar:?] at org.springframework.ai.transformers.TransformersEmbeddingClient.call(TransformersEmbeddingClient.java:280) ~[spring-ai-transformers-1.0.0.jar:1.0.0] at org.springframework.ai.transformers.TransformersEmbeddingClient.embed(TransformersEmbeddingClient.java:232) ~[spring-ai-transformers-1.0.0.jar:1.0.0]

I think this is a bug in the interaction of Spring AI and DJL as both of them shouldn't initialize the ORT environment. It's initialized here in Spring AI (https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-transformers/src/main/java/org/springframework/ai/transformers/TransformersEmbeddingModel.java#L186) and then re-inited with a thread pool in DJL (https://github.com/deepjavalibrary/djl/blob/master/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtEngine.java#L61). The error comes out of ORT as we can't recreate the environment with a different thread pool after it's been initially created due to constraints in the ORT native library.

@tzolov Why only TransformersEmbeddingClient implements InitializingBean and not others? Can you provide some help? thx.

csterwa commented 1 week ago

@brucewkz is this still an issue with more recent versions of Spring AI?