Closed Tql-ws1 closed 1 month ago
@mertalev this is the second report I've seen of ML hwaccel on cuda failing with a segfault. Any chance we have a regression here?
We recently updated to ONNX Runtime 1.18.0. It could be a regression there.
Can you confirm if doing a search (with text in the context field) also causes a segmentation fault? Does face detection cause one? Trying to narrow down whether it's a model-specific behavior.
Can you confirm if doing a search (with text in the context field) also causes a segmentation fault? Does face detection cause one? Trying to narrow down whether it's a model-specific behavior.
The context search and face detection features are working fine, but this is under the condition of solely using CPU for machine learning tasks. I believe it has nothing to do with the model; I've tried models like XLM-Roberta-Large-Vit-B-32, XLM-Roberta-Large-Vit-B-16Plus, and ViT-B-32__openai. If a machine learning task hasn't been successfully executed since creating an immich Docker instance, then I can't even use the context search feature at all.
Here is the log when I used the XLM-Roberta-Large-Vit-B-16Plus model for context search:
[immich-server] | [Nest] 12 - 06/17/2024, 4:19:03 PM LOG [Api:NestApplication] Nest application successfully started
[immich-server] | [Nest] 12 - 06/17/2024, 4:19:03 PM LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:3001 [v1.106.4] [PRODUCTION]
[immich-server] | [Nest] 12 - 06/17/2024, 4:19:04 PM LOG [Api:EventRepository] Websocket Connect: WmHRRT1FjXEXem1aAAAB
[immich-machine-learning] | [06/17/24 08:19:04] INFO Started server process [5]
[immich-machine-learning] | [06/17/24 08:19:04] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:19:04] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:19:04] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:19:04] INFO Application startup complete.
[immich-server] | [Nest] 12 - 06/17/2024, 4:19:58 PM LOG [Api:EventRepository] Websocket Disconnect: WmHRRT1FjXEXem1aAAAB
[immich-server] | [Nest] 12 - 06/17/2024, 4:19:59 PM LOG [Api:EventRepository] Websocket Connect: OmwWXmo9PSh-SNSBAAAD
[immich-machine-learning] | [06/17/24 08:27:46] INFO Setting 'XLM-Roberta-Large-Vit-B-16Plus' execution
[immich-machine-learning] | providers to ['CUDAExecutionProvider',
[immich-machine-learning] | 'CPUExecutionProvider'], in descending order of
[immich-machine-learning] | preference
[immich-machine-learning] | [06/17/24 08:27:46] INFO Loading textual model
[immich-machine-learning] | 'XLM-Roberta-Large-Vit-B-16Plus' to memory
[immich-server] | [Nest] 12 - 06/17/2024, 4:27:52 PM ERROR [Api:Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
[immich-server] | at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:96:27)~xnsl29o7] Failed to search smart
[immich-server] | [Nest] 12 - 06/17/2024, 4:27:52 PM ERROR [Api:Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
[immich-server] | at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:96:27)~xnsl29o7] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-machine-learning] | [06/17/24 08:27:52] ERROR Worker (pid:5) was sent code 139!
[immich-machine-learning] | [06/17/24 08:27:52] INFO Booting worker with pid: 38
[immich-machine-learning] | [06/17/24 08:27:57] INFO Started server process [38]
[immich-machine-learning] | [06/17/24 08:27:57] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:27:57] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:27:57] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:27:57] INFO Application startup complete.
ViT-B-32__openai:
[immich-server] | [Nest] 12 - 06/17/2024, 4:31:47 PM LOG [Api:SystemConfigService~1z3bk5aw] LogLevel=log (set via system config)
[immich-server] | [Nest] 12 - 06/17/2024, 4:31:47 PM LOG [Api:SystemConfigService~1z3bk5aw] LogLevel=log (set via system config)
[immich-server] | [Nest] 2 - 06/17/2024, 4:31:47 PM LOG [Microservices:SystemConfigService] LogLevel=log (set via system config)
[immich-server] | [Nest] 2 - 06/17/2024, 4:31:47 PM LOG [Microservices:MapRepository] Initializing metadata repository
[immich-server] | [Nest] 2 - 06/17/2024, 4:31:47 PM LOG [Microservices:MetadataService] Initialized local reverse geocoder
[immich-server] | [Nest] 12 - 06/17/2024, 4:31:47 PM LOG [Api:SearchRepository~1z3bk5aw] Dimension size of model ViT-B-32__openai is 512, but database expects 640.
[immich-server] | [Nest] 12 - 06/17/2024, 4:31:47 PM LOG [Api:SearchRepository~1z3bk5aw] Updating database CLIP dimension size to 512.
[immich-server] | [Nest] 12 - 06/17/2024, 4:31:49 PM LOG [Api:SearchRepository~1z3bk5aw] Successfully updated database CLIP dimension size from 640 to 512.
[immich-machine-learning] | [06/17/24 08:31:53] INFO Setting 'ViT-B-32__openai' execution providers to
[immich-machine-learning] | ['CUDAExecutionProvider', 'CPUExecutionProvider'],
[immich-machine-learning] | in descending order of preference
[immich-machine-learning] | [06/17/24 08:31:53] INFO Loading textual model 'ViT-B-32__openai' to memory
[immich-server] | [Nest] 12 - 06/17/2024, 4:32:03 PM ERROR [Api:Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
[immich-server] | at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:96:27)~1o4h80vn] Failed to search smart
[immich-server] | [Nest] 12 - 06/17/2024, 4:32:03 PM ERROR [Api:Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
[immich-server] | at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:96:27)~1o4h80vn] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-machine-learning] | [06/17/24 08:32:03] ERROR Worker (pid:38) was sent code 139!
[immich-machine-learning] | [06/17/24 08:32:03] INFO Booting worker with pid: 69
[immich-machine-learning] | [06/17/24 08:32:06] INFO Started server process [69]
[immich-machine-learning] | [06/17/24 08:32:06] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Application startup complete.
Facial detection, buffalo_l model:
[immich-machine-learning] | [06/17/24 08:32:03] ERROR Worker (pid:38) was sent code 139!
[immich-machine-learning] | [06/17/24 08:32:03] INFO Booting worker with pid: 69
[immich-machine-learning] | [06/17/24 08:32:06] INFO Started server process [69]
[immich-machine-learning] | [06/17/24 08:32:06] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:32:06] INFO Application startup complete.
[immich-machine-learning] | [06/17/24 08:33:35] INFO Setting 'buffalo_l' execution providers to
[immich-machine-learning] | ['CUDAExecutionProvider', 'CPUExecutionProvider'],
[immich-machine-learning] | in descending order of preference
[immich-machine-learning] | [06/17/24 08:33:35] INFO Loading detection model 'buffalo_l' to memory
[immich-machine-learning] | [06/17/24 08:33:39] ERROR Worker (pid:69) was sent code 139!
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "4aed4c9a-5fe8-4c23-af7a-bf6daedb5a08"
[immich-server] | }
[immich-server] |
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:39 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "4757da8c-eb74-4bcd-bfca-bceb34dc4f31"
[immich-server] | }
[immich-server] |
[immich-machine-learning] | [06/17/24 08:33:39] INFO Booting worker with pid: 102
[immich-machine-learning] | [06/17/24 08:33:43] INFO Started server process [102]
[immich-machine-learning] | [06/17/24 08:33:43] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:33:43] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:33:43] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:33:43] INFO Application startup complete.
[immich-machine-learning] | [06/17/24 08:33:43] INFO Setting 'buffalo_l' execution providers to
[immich-machine-learning] | ['CUDAExecutionProvider', 'CPUExecutionProvider'],
[immich-machine-learning] | in descending order of preference
[immich-machine-learning] | [06/17/24 08:33:43] INFO Loading detection model 'buffalo_l' to memory
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-machine-learning] | [06/17/24 08:33:47] ERROR Worker (pid:102) was sent code 139!
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "1cef40dc-88e1-4bd6-9fcd-35800a59fb04"
[immich-server] | }
[immich-server] |
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "ea4214f3-c3a0-476a-8dbf-fd5b4ed84b67"
[immich-server] | }
[immich-server] |
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:33:47 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "61c97d5a-f556-439e-a27e-64b5749d4d2f"
[immich-server] | }
[immich-server] |
[immich-machine-learning] | [06/17/24 08:33:47] INFO Booting worker with pid: 137
[immich-machine-learning] | [06/17/24 08:33:51] INFO Started server process [137]
[immich-machine-learning] | [06/17/24 08:33:51] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:33:51] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:33:51] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:33:51] INFO Application startup complete.
antelopev2:
[immich-machine-learning] | [06/17/24 08:33:51] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:33:51] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:33:51] INFO Application startup complete.
[immich-server] | [Nest] 12 - 06/17/2024, 4:36:05 PM LOG [Api:SystemConfigService~sjhnnx0h] LogLevel=log (set via system config)
[immich-server] | [Nest] 12 - 06/17/2024, 4:36:05 PM LOG [Api:SystemConfigService~sjhnnx0h] LogLevel=log (set via system config)
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:05 PM LOG [Microservices:SystemConfigService] LogLevel=log (set via system config)
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:05 PM LOG [Microservices:MapRepository] Initializing metadata repository
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:05 PM LOG [Microservices:MetadataService] Initialized local reverse geocoder
[immich-machine-learning] | [06/17/24 08:36:47] INFO Setting 'antelopev2' execution providers to
[immich-machine-learning] | ['CUDAExecutionProvider', 'CPUExecutionProvider'],
[immich-machine-learning] | in descending order of preference
[immich-machine-learning] | [06/17/24 08:36:47] INFO Loading detection model 'antelopev2' to memory
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "4aed4c9a-5fe8-4c23-af7a-bf6daedb5a08"
[immich-server] | }
[immich-server] |
[immich-machine-learning] | [06/17/24 08:36:51] ERROR Worker (pid:137) was sent code 139!
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error:
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learni
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:51 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "4757da8c-eb74-4bcd-bfca-bceb34dc4f31"
[immich-server] | }
[immich-server] |
[immich-machine-learning] | [06/17/24 08:36:51] INFO Booting worker with pid: 170
[immich-machine-learning] | [06/17/24 08:36:55] INFO Started server process [170]
[immich-machine-learning] | [06/17/24 08:36:55] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:36:55] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:36:55] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:36:55] INFO Application startup complete.
[immich-machine-learning] | [06/17/24 08:36:55] INFO Setting 'antelopev2' execution providers to
[immich-machine-learning] | ['CUDAExecutionProvider', 'CPUExecutionProvider'],
[immich-machine-learning] | in descending order of preference
[immich-machine-learning] | [06/17/24 08:36:55] INFO Loading detection model 'antelopev2' to memory
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:59 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:59 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
[immich-server] | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
[immich-server] | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
[immich-server] | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
[immich-server] | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:274:52)
[immich-server] | at async /usr/src/app/dist/services/job.service.js:148:36
[immich-server] | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[immich-server] | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[immich-server] | [Nest] 2 - 06/17/2024, 4:36:59 PM ERROR [Microservices:JobService] Object:
[immich-server] | {
[immich-server] | "id": "1cef40dc-88e1-4bd6-9fcd-35800a59fb04"
[immich-server] | }
[immich-server] |
[immich-machine-learning] | [06/17/24 08:36:59] ERROR Worker (pid:170) was sent code 139!
[immich-machine-learning] | [06/17/24 08:36:59] INFO Booting worker with pid: 201
[immich-machine-learning] | [06/17/24 08:37:03] INFO Started server process [201]
[immich-machine-learning] | [06/17/24 08:37:03] INFO Waiting for application startup.
[immich-machine-learning] | [06/17/24 08:37:03] INFO Created in-memory cache with unloading after 300s
[immich-machine-learning] | of inactivity.
[immich-machine-learning] | [06/17/24 08:37:03] INFO Initialized request thread pool with 12 threads.
[immich-machine-learning] | [06/17/24 08:37:03] INFO Application startup complete.
I suspect it's an issue with the podman-compose here. This problem was supposedly fixed in commit 79865c2, but the latest version of podman-compose hasn't incorporated this fix yet (as can be seen from the merge time of the PR and the release time of podman-compose v1.1.0). If that's indeed the case, I'm sorry for taking up your valuable time.
I had the same problem with podman-compose v1.1.0 but can confirm now that it works perfectly fine with podman-compose v1.2.0 :tada: Here is the release: https://github.com/containers/podman-compose/releases/tag/v1.2.0
The bug
When attempting to speed up machine learning tasks using CUDA, the 'immich-machine-learning' reports an error as follows: 'Worker (pid:5) was sent code 139!'
The 'immich-server' is indicating errors like this: "ERROR [Microservices:JobService] Unable to run job handler (smartSearch/smart-search): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed."
The OS that Immich Server is running on
Arch
Version of Immich Server
v1.106.4
Version of Immich Mobile App
N/A
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Relevant log output
Additional information
The prerequisites for using CUDA to accelerate machine learning tasks are satisfied. However, there is an issue when hardware acceleration is not being used (using the default configuration in docker-compose.yml) as no such problem arises.