search

codeproject / CodeProject.AI-Server

CodeProject.AI Server is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI.
Other
565 stars 135 forks source link

No Mater what version I use, Constant shut down and restart cycle. #89

Open Strangedave opened 4 months ago

Strangedave commented 4 months ago

Area of Concern

Describe the bug For the last year or so I had been running YOLOv5-6.2 with out much issue. I did see an occasional stoppage, but usually after many weeks of uptime. After installing Server version 2.5.1 I was experiencing frequent stopages, on some occasions the system would restart, but that was not most of the time. After many hours of trouble shooting and reading of online forums, I decided to just flush the system and start over. After reinstalling Blue Iris 5.8.6.6 and CPAI 2.5.1 I was getting about 1/2 a day of uptime, but didnt really have the ambition to chase the problem any more. Today I received an update notification, and thought thank god, hopefully someone solved the issue. After installing the update to 2.5.4 and Blue Iris 5.8.6.7 I have tried 3 modules YOLOv5-6.2 ,YOLOv5 .NET, YOLOv8. all are restarting frequently. YOLOv5.net has been the most stable unfortunately, not catching most things.

Expected behavior Just want the system to stay running.

LOG 15:39:26:System: Windows 15:39:26:Operating System: Windows (Microsoft Windows 11 version 10.0.22631) 15:39:26:CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel) 15:39:26: 1 CPU x 10 cores. 20 logical processors (x64) 15:39:26:GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA) 15:39:26: Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: 15:39:26:System RAM: 48 GiB 15:39:26:Platform: Windows 15:39:26:BuildConfig: Release 15:39:26:Execution Env: Native 15:39:26:Runtime Env: Production 15:39:26:.NET framework: .NET 7.0.15 15:39:26:Default Python: 15:39:26:App DataDir: C:\ProgramData\CodeProject\AI 15:39:26:Video adapter info: 15:39:26: NVIDIA GeForce RTX 3060: 15:39:26: Driver Version 31.0.15.5123 15:39:26: Video Processor NVIDIA GeForce RTX 3060 15:39:26:STARTING CODEPROJECT.AI SERVER 15:39:26:RUNTIMES_PATH = C:\Program Files\CodeProject\AI\runtimes 15:39:26:PREINSTALLED_MODULES_PATH = C:\Program Files\CodeProject\AI\preinstalled-modules 15:39:26:MODULES_PATH = C:\Program Files\CodeProject\AI\modules 15:39:26:PYTHON_PATH = \bin\windows\%PYTHON_NAME%\venv\Scripts\python 15:39:26:Data Dir = C:\ProgramData\CodeProject\AI 15:39:26:Server version: 2.5.4 15:39:29: 15:39:29:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2) 15:39:29:Valid: True 15:39:29:Module Path: \modules\ObjectDetectionYOLOv5-6.2 15:39:29:AutoStart: True 15:39:29:Queue: objectdetection_queue 15:39:29:Runtime: python3.7 15:39:29:Runtime Loc: Shared 15:39:29:FilePath: detect_adapter.py 15:39:29:Pre installed: False 15:39:29:Start pause: 1 sec 15:39:29:Parallelism: 0 15:39:29:LogVerbosity: 15:39:29:Platforms: all,!raspberrypi,!jetson 15:39:29:GPU Libraries: installed if available 15:39:29:GPU Enabled: enabled 15:39:29:Accelerator: 15:39:29:Half Precis.: enable 15:39:29:Environment Variables 15:39:29:APPDIR = \modules\ObjectDetectionYOLOv5-6.2 15:39:29:CPAI_MODULE_ENABLE_GPU = True 15:39:29:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models 15:39:29:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets 15:39:29:MODEL_SIZE = Medium 15:39:29:USE_CUDA = True 15:39:29:YOLOv5_AUTOINSTALL = false 15:39:29:YOLOv5_VERBOSE = false 15:39:29: 15:39:29:Started Object Detection (YOLOv5 6.2) module 15:39:32:Server: This is the latest version 15:39:35:Sending shutdown request to python/ObjectDetectionYOLOv5-6.2 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...12765c) ['No objects found'] took 2167ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6bd0e4) ['No objects found'] took 2167ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e408c) ['No objects found'] took 2175ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...179c35) ['No objects found'] took 2162ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e1103) ['No objects found'] took 2196ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a8a28d) ['No objects found'] took 2206ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ba4cfb) ['No objects found'] took 2215ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6d5838) ['No objects found'] took 2231ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...02c90a) ['No objects found'] took 2230ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6f6fe8) ['No objects found'] took 2246ms 15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d6e3a6) ['No objects found'] took 140ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...23e424) ['No objects found'] took 527ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...0167ac) ['No objects found'] took 515ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...77ed8a) ['No objects found'] took 534ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...6f0dde) ['No objects found'] took 534ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...cbb408) ['No objects found'] took 525ms 15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...5f4e6c) ['No objects found'] took 525ms 15:39:38:detect_adapter.py: Object Detection (YOLOv5 6.2) started. 15:39:38:detect_adapter.py: GPU compute capability is 8.6 15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: GPU compute capability is 8.6 15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060' 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Detecting using ipcam-combined 15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom' 15:39:38:detect_adapter.py: Model Path is C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv5-6.2\custom-models\ipcam-combined.pt 15:39:38:Module ObjectDetectionYOLOv5-6.2 has shutdown 15:39:38:detect_adapter.py: has exited 15:40:08:ObjectDetectionYOLOv5-6.2 went quietly 15:40:08: 15:40:08:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2) 15:40:08:Valid: True 15:40:08:Module Path: \modules\ObjectDetectionYOLOv5-6.2 15:40:08:AutoStart: True 15:40:08:Queue: objectdetection_queue 15:40:08:Runtime: python3.7 15:40:08:Runtime Loc: Shared 15:40:08:FilePath: detect_adapter.py 15:40:08:Pre installed: False 15:40:08:Start pause: 1 sec 15:40:08:Parallelism: 0 15:40:08:LogVerbosity: 15:40:08:Platforms: all,!raspberrypi,!jetson 15:40:08:GPU Libraries: installed if available 15:40:08:GPU Enabled: enabled 15:40:08:Accelerator: 15:40:08:Half Precis.: enable 15:40:08:Environment Variables 15:40:08:APPDIR = \modules\ObjectDetectionYOLOv5-6.2 15:40:08:CPAI_MODULE_ENABLE_GPU = True 15:40:08:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models 15:40:08:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets 15:40:08:MODEL_SIZE = Medium 15:40:08:USE_CUDA = True 15:40:08:YOLOv5_AUTOINSTALL = false 15:40:08:YOLOv5_VERBOSE = false 15:40:08: 15:40:08:Started Object Detection (YOLOv5 6.2) module 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...45f723) ['No objects found'] took 2172ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...129b4c) ['No objects found'] took 2203ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...c16f02) ['No objects found'] took 2194ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...645e6b) ['No objects found'] took 2186ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6e9f5c) ['No objects found'] took 2214ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...360f78) ['No objects found'] took 2246ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e2447e) ['No objects found'] took 2244ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d7ef76) ['No objects found'] took 2283ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...62c31d) ['No objects found'] took 2274ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...062494) ['No objects found'] took 2291ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...0ef1b3) ['No objects found'] took 93ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e9a6d) ['No objects found'] took 110ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...3c126f) ['No objects found'] took 325ms 15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...ffbe51) ['No objects found'] took 525ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...38d0a9) ['No objects found'] took 508ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...e80d0d) ['No objects found'] took 347ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...669b9a) ['No objects found'] took 494ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...7d4ce5) ['No objects found'] took 478ms 15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...07b60f) ['No objects found'] took 473ms 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...49a13c) ['No objects found'] took 61ms 15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...980b04) ['No objects found'] took 50ms 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a7a040) ['No objects found'] took 66ms 15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9debaa) ['No objects found'] took 31ms 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92b1cf) ['No objects found'] took 36ms 15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...66a9e0) ['No objects found'] took 30ms 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...7eed63) ['No objects found'] took 43ms 15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...12bb88) ['No objects found'] took 34ms 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ab823e) ['No objects found'] took 44ms 15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...1bd57d) ['No objects found'] took 31ms 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d1739f) ['No objects found'] took 68ms 15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...20250c) ['No objects found'] took 41ms 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92904a) ['No objects found'] took 87ms 15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9a8375) ['No objects found'] took 41ms 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...cb51b0) ['No objects found'] took 122ms 15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...74185d) ['No objects found'] took 44ms 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e1b343) ['No objects found'] took 75ms 15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8d8369) ['No objects found'] took 37ms 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6cf9dc) ['No objects found'] took 73ms 15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...a4f65e) ['No objects found'] took 48ms 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...8b87bb) ['No objects found'] took 75ms 15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...c83114) ['No objects found'] took 36ms 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...1295e3) ['No objects found'] took 74ms 15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined 15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8c62ae) ['No objects found'] took 39ms

YOLOv8 Might be a different issue 18:50:31:Started Object Detection (YOLOv8) module 18:50:31:detect_adapter.py: Traceback (most recent call last): 18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8\detect_adapter.py", line 12, in 18:50:31:detect_adapter.py: from request_data import RequestData 18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8../../SDK/Python\request_data.py", line 8, in 18:50:31:detect_adapter.py: from PIL import Image 18:50:31:detect_adapter.py: ModuleNotFoundError: No module named 'PIL'

Your System (please complete the following information): Server version: 2.5.4 System: Windows Operating System: Windows (Microsoft Windows 11 version 10.0.22631) CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel) 1 CPU x 10 cores. 20 logical processors (x64) GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA) Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: System RAM: 48 GiB Platform: Windows BuildConfig: Release Execution Env: Native Runtime Env: Production .NET framework: .NET 7.0.15 Default Python: Video adapter info: NVIDIA GeForce RTX 3060: Driver Version 31.0.15.5123 Video Processor NVIDIA GeForce RTX 3060 System GPU info: GPU 3D Usage 0% GPU RAM Usage 1.6 GiB Global Environment variables: CPAI_APPROOTPATH = CPAI_PORT = 32168

Additional context Add any other context about the problem here.

onlineapps-cloud commented 3 months ago

i have same issue on windows.

Strangedave commented 3 months ago

It sems like the most recent update is better. maybe give that a try

onlineapps-cloud commented 3 months ago

It sems like the most recent update is better. maybe give that a try

no, i tried yesterday with latest installer.

ChrisMaunder commented 2 months ago

Are you using Blue Iris? If so, stop Blue Iris from starting/stopping the CodeProject.AI service

onlineapps-cloud commented 2 months ago

here is fixed codeproject ai windows version: https://mega.nz/folder/YzgiEIxC#spjBLYks3nNUrdiQkBquyg proof: image my build will install all neded components, do all exactly as in Readme, and you will obtain working version of codeproject. Soon i will release fixed linux version of codeproject ai.

Strangedave commented 2 months ago

Will there be a 2.6.3 version with the patch included? My setup is working most of the time on 2.6.2, and I don’t really want to mess with a working thing. My system has been online for 118,479 objects, which is probably a week or so. Running YOLOv8. Was looking for a last restart time stamp, but I don’t see that anywhere. Might be a useful add on.