search

codeproject / CodeProject.AI-Server

CodeProject.AI Server is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI.
https://codeproject.github.io/codeproject.ai
Other
723 stars 158 forks source link

No Mater what version I use, Constant shut down and restart cycle. #89

Open Strangedave opened 9 months ago

Strangedave commented 9 months ago

Area of Concern

Describe the bug For the last year or so I had been running YOLOv5-6.2 with out much issue. I did see an occasional stoppage, but usually after many weeks of uptime. After installing Server version 2.5.1 I was experiencing frequent stopages, on some occasions the system would restart, but that was not most of the time. After many hours of trouble shooting and reading of online forums, I decided to just flush the system and start over. After reinstalling Blue Iris 5.8.6.6 and CPAI 2.5.1 I was getting about 1/2 a day of uptime, but didnt really have the ambition to chase the problem any more. Today I received an update notification, and thought thank god, hopefully someone solved the issue. After installing the update to 2.5.4 and Blue Iris 5.8.6.7 I have tried 3 modules YOLOv5-6.2 ,YOLOv5 .NET, YOLOv8. all are restarting frequently. YOLOv5.net has been the most stable unfortunately, not catching most things.

Expected behavior Just want the system to stay running.

LOG 15:39:26:System: Windows 15:39:26:Operating System: Windows (Microsoft Windows 11 version 10.0.22631) 15:39:26:CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel) 15:39:26: 1 CPU x 10 cores. 20 logical processors (x64) 15:39:26:GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA) 15:39:26: Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: 15:39:26:System RAM: 48 GiB 15:39:26:Platform: Windows 15:39:26:BuildConfig: Release 15:39:26:Execution Env: Native 15:39:26:Runtime Env: Production 15:39:26:.NET framework: .NET 7.0.15 15:39:26:Default Python: 15:39:26:App DataDir: C:\ProgramData\CodeProject\AI 15:39:26:Video adapter info: 15:39:26: NVIDIA GeForce RTX 3060: 15:39:26: Driver Version 31.0.15.5123 15:39:26: Video Processor NVIDIA GeForce RTX 3060 15:39:26:STARTING CODEPROJECT.AI SERVER 15:39:26:RUNTIMES_PATH = C:\Program Files\CodeProject\AI\runtimes 15:39:26:PREINSTALLED_MODULES_PATH = C:\Program Files\CodeProject\AI\preinstalled-modules 15:39:26:MODULES_PATH = C:\Program Files\CodeProject\AI\modules 15:39:26:PYTHON_PATH = \bin\windows\%PYTHON_NAME%\venv\Scripts\python 15:39:26:Data Dir = C:\ProgramData\CodeProject\AI 15:39:26:Server version: 2.5.4 15:39:29: 15:39:29:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2) 15:39:29:Valid: True 15:39:29:Module Path: \modules\ObjectDetectionYOLOv5-6.2 15:39:29:AutoStart: True 15:39:29:Queue: objectdetection_queue 15:39:29:Runtime: python3.7 15:39:29:Runtime Loc: Shared 15:39:29:FilePath: detect_adapter.py 15:39:29:Pre installed: False 15:39:29:Start pause: 1 sec 15:39:29:Parallelism: 0 15:39:29:LogVerbosity: 15:39:29:Platforms: all,!raspberrypi,!jetson 15:39:29:GPU Libraries: installed if available 15:39:29:GPU Enabled: enabled 15:39:29:Accelerator: 15:39:29:Half Precis.: enable 15:39:29:Environment Variables 15:39:29:APPDIR = \modules\ObjectDetectionYOLOv5-6.2 15:39:29:CPAI_MODULE_ENABLE_GPU = True 15:39:29:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models 15:39:29:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets 15:39:29:MODEL_SIZE = Medium 15:39:29:USE_CUDA = True 15:39:29:YOLOv5_AUTOINSTALL = false 15:39:29:YOLOv5_VERBOSE = false 15:39:29: 15:39:29:Started Object Detection (YOLOv5 6.2) module 15:39:32:Server: This is the latest version 15:39:35:Sending shutdown request to python/ObjectDetectionYOLOv5-6.2 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...12765c) ['No objects found'] took 2167ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6bd0e4) ['No objects found'] took 2167ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e408c) ['No objects found'] took 2175ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...179c35) ['No objects found'] took 2162ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e1103) ['No objects found'] took 2196ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a8a28d) ['No objects found'] took 2206ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ba4cfb) ['No objects found'] took 2215ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6d5838) ['No objects found'] took 2231ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...02c90a) ['No objects found'] took 2230ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6f6fe8) ['No objects found'] took 2246ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d6e3a6) ['No objects found'] took 140ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...23e424) ['No objects found'] took 527ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...0167ac) ['No objects found'] took 515ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...77ed8a) ['No objects found'] took 534ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...6f0dde) ['No objects found'] took 534ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...cbb408) ['No objects found'] took 525ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...5f4e6c) ['No objects found'] took 525ms 15:39:38:detect_adapter.py: Object Detection (YOLOv5 6.2) started. 15:39:38:detect_adapter.py: GPU compute capability is 8.6 15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: GPU compute capability is 8.6 15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Model Path is C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv5-6.2\custom-models\ipcam-combined.pt 15:39:38:Module ObjectDetectionYOLOv5-6.2 has shutdown 15:39:38:detect_adapter.py: has exited 15:40:08:ObjectDetectionYOLOv5-6.2 went quietly 15:40:08: 15:40:08:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2) 15:40:08:Valid: True 15:40:08:Module Path: \modules\ObjectDetectionYOLOv5-6.2 15:40:08:AutoStart: True 15:40:08:Queue: objectdetection_queue 15:40:08:Runtime: python3.7 15:40:08:Runtime Loc: Shared 15:40:08:FilePath: detect_adapter.py 15:40:08:Pre installed: False 15:40:08:Start pause: 1 sec 15:40:08:Parallelism: 0 15:40:08:LogVerbosity: 15:40:08:Platforms: all,!raspberrypi,!jetson 15:40:08:GPU Libraries: installed if available 15:40:08:GPU Enabled: enabled 15:40:08:Accelerator: 15:40:08:Half Precis.: enable 15:40:08:Environment Variables 15:40:08:APPDIR = \modules\ObjectDetectionYOLOv5-6.2 15:40:08:CPAI_MODULE_ENABLE_GPU = True 15:40:08:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models 15:40:08:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets 15:40:08:MODEL_SIZE = Medium 15:40:08:USE_CUDA = True 15:40:08:YOLOv5_AUTOINSTALL = false 15:40:08:YOLOv5_VERBOSE = false 15:40:08: 15:40:08:Started Object Detection (YOLOv5 6.2) module 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...45f723) ['No objects found'] took 2172ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...129b4c) ['No objects found'] took 2203ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...c16f02) ['No objects found'] took 2194ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...645e6b) ['No objects found'] took 2186ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6e9f5c) ['No objects found'] took 2214ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...360f78) ['No objects found'] took 2246ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e2447e) ['No objects found'] took 2244ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d7ef76) ['No objects found'] took 2283ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...62c31d) ['No objects found'] took 2274ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...062494) ['No objects found'] took 2291ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...0ef1b3) ['No objects found'] took 93ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e9a6d) ['No objects found'] took 110ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...3c126f) ['No objects found'] took 325ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...ffbe51) ['No objects found'] took 525ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...38d0a9) ['No objects found'] took 508ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...e80d0d) ['No objects found'] took 347ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...669b9a) ['No objects found'] took 494ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...7d4ce5) ['No objects found'] took 478ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...07b60f) ['No objects found'] took 473ms 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...49a13c) ['No objects found'] took 61ms 15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...980b04) ['No objects found'] took 50ms 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a7a040) ['No objects found'] took 66ms 15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9debaa) ['No objects found'] took 31ms 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92b1cf) ['No objects found'] took 36ms 15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...66a9e0) ['No objects found'] took 30ms 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...7eed63) ['No objects found'] took 43ms 15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...12bb88) ['No objects found'] took 34ms 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ab823e) ['No objects found'] took 44ms 15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...1bd57d) ['No objects found'] took 31ms 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d1739f) ['No objects found'] took 68ms 15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...20250c) ['No objects found'] took 41ms 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92904a) ['No objects found'] took 87ms 15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9a8375) ['No objects found'] took 41ms 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...cb51b0) ['No objects found'] took 122ms 15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...74185d) ['No objects found'] took 44ms 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e1b343) ['No objects found'] took 75ms 15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8d8369) ['No objects found'] took 37ms 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6cf9dc) ['No objects found'] took 73ms 15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...a4f65e) ['No objects found'] took 48ms 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...8b87bb) ['No objects found'] took 75ms 15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...c83114) ['No objects found'] took 36ms 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...1295e3) ['No objects found'] took 74ms 15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8c62ae) ['No objects found'] took 39ms

YOLOv8 Might be a different issue 18:50:31:Started Object Detection (YOLOv8) module 18:50:31:detect_adapter.py: Traceback (most recent call last): 18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8\detect_adapter.py", line 12, in 18:50:31:detect_adapter.py: from request_data import RequestData 18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8../../SDK/Python\request_data.py", line 8, in 18:50:31:detect_adapter.py: from PIL import Image 18:50:31:detect_adapter.py: ModuleNotFoundError: No module named 'PIL'

Your System (please complete the following information): Server version: 2.5.4 System: Windows Operating System: Windows (Microsoft Windows 11 version 10.0.22631) CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel) 1 CPU x 10 cores. 20 logical processors (x64) GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA) Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: System RAM: 48 GiB Platform: Windows BuildConfig: Release Execution Env: Native Runtime Env: Production .NET framework: .NET 7.0.15 Default Python: Video adapter info: NVIDIA GeForce RTX 3060: Driver Version 31.0.15.5123 Video Processor NVIDIA GeForce RTX 3060 System GPU info: GPU 3D Usage 0% GPU RAM Usage 1.6 GiB Global Environment variables: CPAI_APPROOTPATH = CPAI_PORT = 32168

Additional context Add any other context about the problem here.

onlineapps-cloud commented 7 months ago

i have same issue on windows.

Strangedave commented 7 months ago

It sems like the most recent update is better. maybe give that a try

onlineapps-cloud commented 7 months ago

It sems like the most recent update is better. maybe give that a try

no, i tried yesterday with latest installer.

ChrisMaunder commented 7 months ago

Are you using Blue Iris? If so, stop Blue Iris from starting/stopping the CodeProject.AI service

onlineapps-cloud commented 7 months ago

here is fixed codeproject ai windows version: https://mega.nz/folder/YzgiEIxC#spjBLYks3nNUrdiQkBquyg proof: image my build will install all neded components, do all exactly as in Readme, and you will obtain working version of codeproject. Soon i will release fixed linux version of codeproject ai.

Strangedave commented 7 months ago

Will there be a 2.6.3 version with the patch included? My setup is working most of the time on 2.6.2, and I don’t really want to mess with a working thing. My system has been online for 118,479 objects, which is probably a week or so. Running YOLOv8. Was looking for a last restart time stamp, but I don’t see that anywhere. Might be a useful add on.