search

marieai / marie-ai

Integrate AI-powered Document Analysis Pipelines
MIT License
60 stars 5 forks source link

Extract error : cache resources exhausted @ error/cache.c/OpenPixelCache/4095 #95

Closed gregbugaj closed 11 months ago

gregbugaj commented 11 months ago

Describe the bug

While running inside the container the application fails with 

ERROR  marie@35 Extract error : cache resources exhausted `/tmp/tmpb0pn1n63/PID_5805_12996_0_191120610_00005.tif' @ error/cache.c/OpenPixelCache/4095            [12/04/23 14:17:25]
       Traceback (most recent call last):                                                                                                                                           
         File "/opt/venv/lib/python3.10/site-packages/marie/executor/classifier/document_classifier_executor.py", line 199, in classify                                             
           metadata = self.pipeline.execute(                                                                                                                                        
         File "/opt/venv/lib/python3.10/site-packages/marie/pipe/classification_pipeline.py", line 213, in execute                                                                  
           return self.execute_frames_pipeline(                                                                                                                                     
         File "/opt/venv/lib/python3.10/site-packages/marie/pipe/classification_pipeline.py", line 137, in execute_frames_pipeline                                                  
           burst_frames(ref_id, frames, root_asset_dir)                                                                                                                             
         File "/opt/venv/lib/python3.10/site-packages/marie/pipe/components.py", line 353, in burst_frames                                                                          
           burst_tiff_frames(frames, output_dir, filename_generator=filename_generator)                                                                                             
         File "/opt/venv/lib/python3.10/site-packages/marie/utils/tiff_ops.py", line 130, in burst_tiff_frames                                                                      
           __process_burst(                                                                                                                                                         
         File "/opt/venv/lib/python3.10/site-packages/marie/utils/tiff_ops.py", line 100, in __process_burst                                                                        
           raise ident                                                                                                                                                              
         File "/opt/venv/lib/python3.10/site-packages/marie/utils/tiff_ops.py", line 98, in __process_burst                                                                         
           convert_group4(output_path_tmp, output_path)                                                                                                                             
         File "/opt/venv/lib/python3.10/site-packages/marie/utils/tiff_ops.py", line 48, in convert_group4                                                                          
           with Image(filename=src_path) as image:                                                                                                                                  
         File "/opt/venv/lib/python3.10/site-packages/wand/image.py", line 9382, in __init__                                                                                        
           self.read(filename=filename)                                                                                                                                             
         File "/opt/venv/lib/python3.10/site-packages/wand/image.py", line 10137, in read                                                                                           
           self.raise_exception()                                                                                                                                                   
         File "/opt/venv/lib/python3.10/site-packages/wand/resource.py", line 225, in raise_exception                                                                               
           raise e                                                                                                                                                                  
       wand.exceptions.CacheError: cache resources exhausted `/tmp/tmpb0pn1n63/PID_5805_12996_0_191120610_00005.tif' @ error/cache.c/OpenPixelCache/4095

Describe how you solve it Issue looks to be related to ImageMagic. We are already providing custom im-policy.xml so there might be some additional tweaking needed there.

gregbugaj commented 11 months ago

Adjusting the resources in the /etc/ImageMagick-6/policy.xml looks like it have solved the problem.

Values have been adjusted to following:

  <policy domain="resource" name="memory" value="32000MiB"/>
  <policy domain="resource" name="map" value="8GiB"/>
  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="8GiB"/>
  <policy domain="resource" name="disk" value="8GiB"/>

This can be verified by running following command identify -list resource

Resource limits:
  Width: 16KP
  Height: 16KP
  List length: unlimited
  Area: 8.5899GP
  Memory: 31.25GiB
  Map: 8GiB
  Disk: 8GiB
  File: 3072
  Thread: 20
  Throttle: 0
  Time: unlimited