search

Shared-Reality-Lab / IMAGE-server

IMAGE project server components
Other
2 stars 7 forks source link

On boot, pegasus not launching all IMAGE containers #776

Closed jeffbl closed 9 months ago

jeffbl commented 10 months ago

Reboot pegasus Make a request for a photo NOTE: no audio renderings, only a "text only" rendering appears docker compose up -d NOTE: multiple additional containers start:

jeffbl@pegasus /v/d/image> docker compose up -d
[+] Running 27/27
 ✔ Container image-object-detection-1             Running                                                                           0.0s 
 ✔ Container image-nominatim-preprocessor-1       Running                                                                           0.0s 
 ✔ Container image-map-tactile-svg-handler-1      Running                                                                           0.0s 
 ✔ Container image-photo-tactile-svg-handler-1    Running                                                                           0.0s 
 ✔ Container image-openstreetmap-1                Running                                                                           0.0s 
 ✔ Container image-semantic-segmentation-1        Started                                                                           0.0s 
 ✔ Container image-object-grouping-1              Running                                                                           0.0s 
 ✔ Container image-graphic-tagger-1               Running                                                                           0.0s 
 ✔ Container image-photo-audio-haptics-handler-1  Running                                                                           0.0s 
 ✔ Container image-supercollider-1                Running                                                                           0.0s 
 ✔ Container image-goaccess                       Running                                                                           0.0s 
 ✔ Container image-content-categoriser-1          Running                                                                           0.0s 
 ✔ Container image-supercollider-2                Running                                                                           0.0s 
 ✔ Container image-object-sorting-1               Running                                                                           0.0s 
 ✔ Container image-svg-od-handler-1               Running                                                                           0.0s 
 ✔ Container image-espnet-tts-1                   Started                                                                           0.0s 
 ✔ Container image-high-charts-handler-1          Running                                                                           0.0s 
 ✔ Container image-autour-preprocessor-1          Running                                                                           0.0s 
 ✔ Container image-photo-audio-handler-1          Running                                                                           0.0s 
 ✔ Container image-autour-handler-1               Running                                                                           0.0s 
 ✔ Container image-multilang-support-1            Started                                                                           0.0s 
 ✔ Container image-svg-depth-map-1                Running                                                                           0.0s 
 ✔ Container image-depth-map-generator-1          Started                                                                           0.0s 
 ✔ Container image-svg-semantic-seg-handler-1     Running                                                                           0.0s 
 ✔ Container image-orchestrator-1                 Running                                                                           0.0s 
 ✔ Container image-espnet-tts-fr-1                Started                                                                           0.0s 
 ✔ Container image-image-pegasus-cim-mcgill-ca-1  Running                                                                           0.0s 
jeffbl@pegasus /v/d/image>

EXPECT: All necessary containers running right after reboot, without need to manually restart things. NOTE: The ones that get started all use GPU?

When making IMAGE request right after reboot, log shows:

image-orchestrator-1                 | 2023-09-29T13:27:59.607082531Z Received request                                                                                                                                                                                   [757/1198]
image-orchestrator-1                 | 2023-09-29T13:27:59.630381461Z Running preprocessors in parallel...                                                                                                                                                                         
image-orchestrator-1                 | 2023-09-29T13:27:59.630515196Z Now on priority group 1                                                                                                                                                                                      
image-orchestrator-1                 | 2023-09-29T13:27:59.630624835Z Sending to preprocessor "content-categoriser"                                                                                                                                                                
image-orchestrator-1                 | 2023-09-29T13:27:59.634192055Z Sending to preprocessor "autour-preprocessor"                                                                                                                                                                
image-content-categoriser-1          | 2023-09-29T13:27:59.636486759Z [2023-09-29 13:27:59 +0000] [9] [DEBUG] POST /preprocessor                                                                                                                                                   
image-autour-preprocessor-1          | 2023-09-29T13:27:59.636754810Z [2023-09-29 13:27:59 +0000] [7] [DEBUG] POST /preprocessor                                                                                                                                                   
image-orchestrator-1                 | 2023-09-29T13:27:59.961481642Z Now on priority group 2                                                                                                                                                                                      
image-orchestrator-1                 | 2023-09-29T13:27:59.961496451Z Sending to preprocessor "nominatim-preprocessor"                                                                                                                                                             
image-orchestrator-1                 | 2023-09-29T13:27:59.962380375Z Sending to preprocessor "graphic-tagger"                                                                                                                                                                     
image-orchestrator-1                 | 2023-09-29T13:27:59.963457459Z Sending to preprocessor "openstreetmap"                                                                                                                                                                      
image-graphic-tagger-1               | 2023-09-29T13:27:59.965033784Z [2023-09-29 13:27:59 +0000] [7] [DEBUG] POST /preprocessor                                                                                                                                                   
image-openstreetmap-1                | 2023-09-29T13:27:59.965151478Z [2023-09-29 13:27:59 +0000] [7] [DEBUG] POST /preprocessor                                                                                                                                                   
image-openstreetmap-1                | 2023-09-29T13:27:59.965551771Z 23-09-29 13:27 UTC [DEBUG]: Received request                                                                                                                                                                 
image-openstreetmap-1                | 2023-09-29T13:27:59.966977910Z 23-09-29 13:27 UTC [DEBUG]: Validating request                                                                                                                                                               
image-openstreetmap-1                | 2023-09-29T13:27:59.968478440Z 23-09-29 13:27 UTC [INFO]: Not map content. Skipping...                                                                                                                                                      
image-nominatim-preprocessor-1       | 2023-09-29T13:27:59.972999427Z Received request                                                                                                                                                                                             
image-nominatim-preprocessor-1       | 2023-09-29T13:27:59.980905397Z Coordinates not available, cannot make a request for reverse geocode.                                                                                                                                        
image-orchestrator-1                 | 2023-09-29T13:28:00.237866319Z Now on priority group 3                                                                                                                                                                                      
image-orchestrator-1                 | 2023-09-29T13:28:00.237878102Z Sending to preprocessor "object-detection"                                                                                                                                                                   
image-object-detection-1             | 2023-09-29T13:28:00.239634018Z [2023-09-29 13:28:00 +0000] [7] [DEBUG] POST /preprocessor                                                                                                                                                   
image-orchestrator-1                 | 2023-09-29T13:28:01.589227980Z Now on priority group 4                                                                                                                                                                                      
image-orchestrator-1                 | 2023-09-29T13:28:01.589246205Z Sending to preprocessor "object-grouping"                                                                                                                                                                    
image-object-grouping-1              | 2023-09-29T13:28:01.591031026Z [2023-09-29 13:28:01 +0000] [8] [DEBUG] POST /preprocessor                                                                                                                                                   
image-orchestrator-1                 | 2023-09-29T13:28:01.594893467Z Now on priority group 5                                                                                                                                                                                      
image-orchestrator-1                 | 2023-09-29T13:28:01.594905240Z Sending to preprocessor "object-sorting"                                                                                                                                                                     
image-object-sorting-1               | 2023-09-29T13:28:01.596686364Z [2023-09-29 13:28:01 +0000] [8] [DEBUG] POST /preprocessor                                                                                                                                                   
image-orchestrator-1                 | 2023-09-29T13:28:01.610275200Z Waiting for handlers...                                                                                                                                                                                      
image-svg-depth-map-1                | 2023-09-29T13:28:01.612088776Z [2023-09-29 13:28:01 +0000] [7] [DEBUG] POST /handler                                                                                                                                                        
image-svg-semantic-seg-handler-1     | 2023-09-29T13:28:01.612390972Z [2023-09-29 13:28:01 +0000] [7] [DEBUG] POST /handler                                                                                                                                                        
image-map-tactile-svg-handler-1      | 2023-09-29T13:28:01.612760056Z [2023-09-29 13:28:01 +0000] [7] [DEBUG] POST /handler                                                                                                                                                        
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.612930430Z [2023-09-29 13:28:01 +0000] [7] [DEBUG] POST /handler                                                                                                                                                        
image-svg-od-handler-1               | 2023-09-29T13:28:01.613242385Z [2023-09-29 13:28:01 +0000] [7] [DEBUG] POST /handler                                                                                                                                                        
image-map-tactile-svg-handler-1      | 2023-09-29T13:28:01.617323773Z DEBUG:root:Received request                                                                                                                                                                                  
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.617422411Z DEBUG:root:Received request                                                                                                                                                                                  
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.619179099Z DEBUG:root:Validating request schema                                                                                                                                                                         
image-map-tactile-svg-handler-1      | 2023-09-29T13:28:01.620013329Z DEBUG:root:TactileSVG Renderer not supported                                                                                                                                                                 
image-map-tactile-svg-handler-1      | 2023-09-29T13:28:01.620074796Z DEBUG:root:Missing 'ca.mcgill.a11y.image.renderer.TactileSVG'. Sending empty response.                                                                                                                       
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.620185276Z DEBUG:root:Checking whether renderer is supported                                                                                                                                                            
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.620193322Z DEBUG:root:TactileSVG Renderer not supported                                                                                                                                                                 
image-photo-tactile-svg-handler-1    | 2023-09-29T13:28:01.620274506Z DEBUG:root:Sending response                                                                                                                                                                                  
image-photo-audio-handler-1          | 2023-09-29T13:28:01.621755599Z Received request                                                                                                                                                                                             
image-high-charts-handler-1          | 2023-09-29T13:28:01.621968585Z Received request                                                                                                                                                                                             
image-photo-audio-haptics-handler-1  | 2023-09-29T13:28:01.623075074Z Received request                                                                                                                                                                                             
image-autour-handler-1               | 2023-09-29T13:28:01.623548095Z Request received!                                                                                                                                                                                            
image-high-charts-handler-1          | 2023-09-29T13:28:01.630053884Z No high charts data in this request. Skipping.                                                                                                                                                               
image-photo-audio-haptics-handler-1  | 2023-09-29T13:28:01.631073016Z Photo audio-haptic renderer not supported!                                                                                                                                                                   
image-autour-handler-1               | 2023-09-29T13:28:01.631415509Z Not enough data to generate a rendering.                                                                                                                                                                     
image-photo-audio-handler-1          | 2023-09-29T13:28:01.631966438Z Rendering title                                                                                                                                                                                              
image-photo-audio-handler-1          | 2023-09-29T13:28:01.631970757Z Constructing text renderings                                                                                                                                                                                 
image-photo-audio-handler-1          | 2023-09-29T13:28:01.632388122Z Generating TTS Response                                                                                                                                                                                      
image-photo-audio-handler-1          | 2023-09-29T13:28:01.632517779Z Getting TTS in "en"                                                                                                                                                                                          
image-photo-audio-handler-1          | 2023-09-29T13:28:06.660008241Z Failed to generate audio!                                                                                                                                                                                    
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661442523Z TypeError: fetch failed                                                                                                                                                                                      
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661452722Z     at Object.fetch (node:internal/deps/undici/undici:11576:11)
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661456019Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661458463Z   cause: Error: getaddrinfo EAI_AGAIN espnet-tts
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661460647Z       at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661471047Z Sending response
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661477630Z     errno: -3001,
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661488130Z     code: 'EAI_AGAIN',
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661491156Z     syscall: 'getaddrinfo',
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661493400Z     hostname: 'espnet-tts'
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661495564Z   }
image-photo-audio-handler-1          | 2023-09-29T13:28:06.661497768Z }
image-orchestrator-1                 | 2023-09-29T13:28:06.673178088Z Valid response generated.
image-image-pegasus-cim-mcgill-ca-1  | 2023-09-29T13:28:06.679164411Z 10.122.190.61 - - [29/Sep/2023:13:28:06 +0000] "POST /render HTTP/1.1" 200 289 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
image-orchestrator-1                 | 2023-09-29T13:28:06.680587703Z Wrote temporary files to /var/log/IMAGE/1f2fdec9-4883-4c56-bd4e-87d0d5357590
image-image-pegasus-cim-mcgill-ca-1  | 2023-09-29T13:28:08.825477625Z 50.17.185.102 - - [29/Sep/2023:13:28:08 +0000] "GET / HTTP/1.1" 200 10351 "-" "FreshpingBot/1.0 (+https://freshping.io/)"
jeffbl commented 10 months ago

We just did system updates, and rebooted again. Everything seems OK on this reboot. Only difference was that this reboot was done with the machine in a working state, whereas the first one was when there were containers in a bad state. Don't know if that is relevant, or is this just random.

jeffbl commented 9 months ago

Pegasus was rebooted again, and we don't see any issues with anything starting, and results seem to come back fine from extension requests. @JRegimbal closing since we cannot reproduce this, can reactivate if we see it again.