Body Tracking Sample doesn't show any characters but just Model Opt failed message

dedoogong commented 1 year ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
[X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

I've run the sample with UE 5.1, VS 2022, ZED 2i(SDK4), CUDA 12. I installed the ZED 2i at the high position(higher than me) and it's located far from me around 2 meters to make it detect floor and me correctly. The project build process was run smoothly without errors. When I run the project, it shows Model Optimization was failed as below; "SlCameraProxy: Warning: Waiting for ZED to be connected SlCameraProxy: Warning: ZED connected SlCameraProxy: Initializing SlCameraProxy: Initialized SlCameraProxy: Warning: Optimizing AI model. The process can take few minutes ... SlAIThread: BT Thread started SlGrabThread: Thread started SlAIThread: Warning: Optimizing AI model. The process can take few minutes SlCameraProxy: Warning: Optim done SlCameraProxy: Error: Error during model optimization" When I run ZED 2i in NEURAL mode with the default ZED SDK Depth Viewer, it successfully optimize the ZED AI models and run in 15FPS. Can you guess what is the missing/wrong point? Please help me. I want to jump into the virtual world as soon as possible!

Thank you!

Steps to Reproduce

Build ZED UE5 Sample Projects with VS 2019
Open the UE5 Project and play the project.

Expected Result

Default Character should track me and UE shows it.

Actual Result

No character was shown

ZED Camera model

ZED2i

Environment

OS: WIN 10
CPU : intel i9
GPU : RTX 2060
ZED SDK : 4.0
UE : 5.1

Anything else?

No response

SLJLMacarit commented 1 year ago

@dedoogong What exact version of the ZED SDK and UE5 project are you using? I mean, including the patch version (4.0.x). We'll investigate this, in the meantime optimizing the models through ZED Diagnostic should work, please optimize them and then try the project again. Procedure detailed here: Optimize ZED SDK AI models manually Basically:

Close applications using your GPU (like UE5)
Go to ZED SDK/tools folder
Run in console: ./"ZED Diagnostic.exe" -aic
Then run: ./"ZED Diagnostic.exe" -aio to optimize all models. It will take a while but needs to be done only once. Refer to the previous link to optimize only specific models.
Check the project runs correctly

dedoogong commented 1 year ago

ZDK version is 4.0.6, UE5 is 5.1.1.

I did all your guides as below;

PS C:\Program Files (x86)\ZED SDK\tools> ./"ZED Diagnostic.exe" -aic

Removing all AI models
PS C:\Program Files (x86)\ZED SDK\tools> ./"ZED Diagnostic.exe" -aio

Optimizing all AI models
Optimizing: MULTI CLASS DETECTION...
 Downloading objects_performance_3.2  100.0%[========================================] 37.6/37.6 MB              Done
 Optimizing objects_performance_3.2 |  100.0%[=============================>] Done ! in 6min 14s
Optimizing: MULTI CLASS MEDIUM DETECTION...
 Downloading objects_medium_3.2  100.0%[========================================] 85.5/85.5 MB              Done
 Optimizing objects_medium_3.2 |  100.0%[=============================>] Done ! in 4min 42s
Optimizing: MULTI CLASS ACCURATE DETECTION...
 Downloading objects_accurate_3.2  100.0%[========================================] 142.7/142.7 MB              Done
 Optimizing objects_accurate_3.2 |  100.0%[=============================>] Done ! in 4min 16s
Optimizing: HUMAN BODY FAST DETECTION...
 Downloading skeleton_body18_3.2  100.0%[========================================] 94.7/94.7 MB              Done
 Optimizing skeleton_body18_3.2 |  100.0%[=============================>] Done ! in 1min 29s
Optimizing: HUMAN BODY MEDIUM DETECTION...
 Optimizing skeleton_body18_3.2 |  100.0%[=============================>] Done ! in 1min 35s
Optimizing: HUMAN BODY ACCURATE DETECTION...
 Optimizing skeleton_body18_3.2 |  100.0%[=============================>] Done ! in 1min 43s
Optimizing: HUMAN BODY 38 FAST DETECTION...
 Downloading skeleton_body38_3.5  100.0%[========================================] 100.8/100.8 MB              Done
 Optimizing skeleton_body38_3.5 |  100.0%[=============================>] Done ! in 0min 45s
Optimizing: HUMAN BODY 38 MEDIUM DETECTION...
 Optimizing skeleton_body38_3.5 |  100.0%[=============================>] Done ! in 0min 49s
Optimizing: HUMAN BODY 38 ACCURATE DETECTION...
 Optimizing skeleton_body38_3.5 |  100.0%[=============================>] Done ! in 0min 49s
Optimizing: PERSON HEAD DETECTION...
 Downloading person_head_performance_2.4  100.0%[========================================] 4.8/4.8 MB              Done
 Optimizing person_head_performance_2.4 |  100.0%[=============================>] Done ! in 2min 49s
Optimizing: PERSON HEAD ACCURATE DETECTION...
 Downloading person_head_accurate_2.4  100.0%[========================================] 56.6/56.6 MB              Done
 Optimizing person_head_accurate_2.4 |  100.0%[=============================>] Done ! in 3min 44s
Optimizing: REID ASSOCIATION...
 Downloading person_reid_1.4  100.0%[========================================] 8.4/8.4 MB              Done
 Optimizing person_reid_1.4 |  100.0%[=============================>] Done ! in 1min 15s
Optimizing: NEURAL DEPTH...
 Downloading neural_depth_2.0  100.0%[========================================] 18.4/18.4 MB              Done
 Optimizing neural_depth_2.0 |  100.0%[=============================>] Done ! in 0min 49s

and this time another error occurs as below;

LogWorld: Bringing World /Game/ZED/Levels/UEDPIE_0_L_BodyTrackingSingle.L_BodyTrackingSingle up for play (max tick rate 0) at 2023.10.02-22.47.57
LogWorld: Bringing up level for play took: 0.000555
LogOnline: OSS: Created online subsystem instance for: :Context_3
PIE: 서버가 로그인했습니다.
PIE: 에디터에서 플레이 총 시작 시간 0.045초입니다.
SlCameraProxy: Warning: Waiting for ZED to be connected
SlCameraProxy: Warning: ZED connected
SlCameraProxy: Initializing
SlCameraProxy: Error: Error during initialization: "EC_CameraFailedToSetup"
SlCameraProxy: Warning: Not initialized

Even though, when I run the ZED depth viewer and it shows the depth correctly so there is no camera setup problem.

dedoogong commented 1 year ago

ah it works now! I didnt connected it to USE 3.0. Thank you very much. By the way, even I set it to HD 2K, accurate body pose inference, the result is little not OK( neck and some joints are slightly twisted ..) I think my room is bright enough. are there some recommended, best physical settings like clothes color or wall/floor color, material, how much large empty space needed and so on ?

Bvallon-sl commented 1 year ago

Yes, the lighting conditions can affect the detection and tracking of the key points. The detection won't be that affected by your cloths or walls colors but the depth estimation can be.

But, as long as you are fully visible in the image (make sure your feet are visible), it should be fine.

stereolabs / zed-UE5