search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
49.36k stars 2.59k forks source link

[BUG] objectTagging/classify-image can't run on apple silicon #4918

Closed Chiunownow closed 10 months ago

Chiunownow commented 11 months ago

The bug

Environment

Issue

ML job objectTagging/classify-image can't run on apple cilicon with microsoft/resnet-50 and apple/mobilevit-small.

Logs

# microservices side

immich_microservices     | [Nest] 7  - 11/09/2023, 2:11:26 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
immich_microservices     | [Nest] 7  - 11/09/2023, 2:11:26 AM   ERROR [JobService] TypeError: fetch failed
immich_microservices     |     at Object.fetch (node:internal/deps/undici/undici:11372:11)
immich_microservices     |     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
immich_microservices     |     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
immich_microservices     |     at async /usr/src/app/dist/domain/job/job.service.js:108:37
immich_microservices     |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
immich_microservices     |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
immich_microservices     | [Nest] 7  - 11/09/2023, 2:11:26 AM   ERROR [JobService] Object:
immich_microservices     | {
immich_microservices     |   "id": "071d0fc4-39d3-4d40-a8cd-f75c715f3396"
immich_microservices     | }
# ML side with microsoft/resnet-50

[11/09/23 02:16:04] INFO     Loading facial recognition model 'buffalo_l'       
[11/09/23 02:16:04] INFO     Loading clip model 'ViT-B-32__openai'              
[11/09/23 02:16:05] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                              
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[11/09/23 02:16:05] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                        
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNextImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
    - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[11/09/23 02:16:15] ERROR    Worker (pid:104) was sent SIGSEGV!  

# with apple/mobilevit-small

[11/09/23 02:29:48] INFO     Loading facial recognition model 'buffalo_l'       
[11/09/23 02:29:48] INFO     Downloading image classification model             
                             'apple/mobilevit-small'.This may take a while.     
[11/09/23 02:29:48] INFO     Loading clip model 'ViT-B-32__openai'              
Downloading (…)rocessor_config.json: 100%|██████████| 187/187 [00:00<00:00, 240kB/s]
Downloading (…)ackage/Manifest.json: 100%|██████████| 617/617 [00:00<00:00, 1.44MB/s]
Downloading (…)aad582ab/config.json: 100%|██████████| 70.0k/70.0k [00:00<00:00, 2.18MB/s]
Downloading pytorch_model.bin: 100%|██████████| 22.5M/22.5M [00:02<00:00, 9.38MB/s]
Downloading weight.bin: 100%|██████████| 22.3M/22.3M [00:02<00:00, 8.91MB/s].5MB/s]
Fetching 5 files: 100%|██████████| 5/5 [00:04<00:00,  1.15it/s]02, 5.75MB/s]
[11/09/23 02:29:54] INFO     Loading image classification model                 
                             'apple/mobilevit-small'                            
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[11/09/23 02:29:54] INFO     ONNX model not found in cache directory for        
                             'apple/mobilevit-small'.Exporting optimized model  
                             for future use.                                    
Downloading (…)lve/main/config.json: 100%|██████████| 70.0k/70.0k [00:00<00:00, 4.70MB/s]
Framework not specified. Using pt to export to ONNX.
Downloading (…)rocessor_config.json: 100%|██████████| 187/187 [00:00<00:00, 413kB/s]
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/mobilevit/feature_extraction_mobilevit.py:28: FutureWarning: The class MobileViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use MobileViTImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
    - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
[11/09/23 02:30:06] ERROR    Worker (pid:17) was sent SIGSEGV!                  
/usr/local/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

The OS that Immich Server is running on

Unraid and M1 MBA

Version of Immich Server

v1.85.0

Version of Immich Mobile App

-

Platform with the issue

Your docker-compose.yml content

version: '3.8'

services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:v1.85.0
    volumes:
      - model-cache:/cache
    restart: always
    ports:
      - 3003:3003

volumes:
  model-cache:

Your .env content

-

Reproduction steps

-

Additional information

No response

alextran1502 commented 11 months ago

Please use the model listed in this page https://huggingface.co/immich-app

Chiunownow commented 11 months ago

Sorry, I don't know which model I should choose. I try immich-app/RN50__cc12m immich-app/RN50x64__openai, they all get error log like

Exception in ASGI application
Traceback (most recent call last):
  File "/usr/src/app/main.py", line 101, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 65, in load
    self._load()
  File "/usr/src/app/models/image_classification.py", line 39, in _load
    processor = AutoImageProcessor.from_pretrained(self.cache_dir, cache_dir=self.cache_dir)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/transformers/models/auto/image_processing_auto.py", line 346, in from_pretrained
    config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/transformers/image_processing_utils.py", line 329, in get_image_processor_dict
    resolved_image_processor_file = cached_file(
                                    ^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/transformers/utils/hub.py", line 400, in cached_file
    raise EnvironmentError(
OSError: /cache/image-classification/immich-app_RN50x64__openai does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co//cache/image-classification/immich-app_RN50x64__openai/None' for available files.
DevinWain commented 11 months ago

Please use the model listed in this page https://huggingface.co/immich-app

I had the same problem, but I can't find any collection for image classification in https://huggingface.co/immich-app.

benjamonnguyen commented 11 months ago

Any updates on this problem?