STAF unable to find BODY_21A

[anilkunchalaece]

Hi, I checkout to the staf branch

* (HEAD detached at origin/staf)
  master

and downloaded the models using ./getModels.sh

├── pose
│   ├── body_21a
│   │   ├── pose_deploy.prototxt
│   │   └── pose_iter_264000.caffemodel
│   ├── body_21a_video
│   │   ├── pose_deploy.prototxt
│   │   └── pose_iter_264000.caffemodel
│   ├── body_25
│   │   ├── pose_deploy.prototxt
│   │   └── pose_iter_584000.caffemodel
│   ├── coco

but I'm still getting the model file not found error **Caffe trained model file not found: models/pose/body_21a/pose_iter_XXXXXX.caffemodel.** any suggestions ? Note : able to run with BODY_25 without any issues.

Originally posted by @anilkunchalaece in https://github.com/mkocabas/VIBE/issues/30#issuecomment-895526901

[anilkunchalaece]

Looks like problem with --tracking parameter. If i use any value (0 or 5) except 1 for tracking I'm getting the file not found error build/examples/openpose/openpose.bin --model_pose BODY_21A --render_pose 1 --tracking 0 --video sample.mp4 --net_resolution=320x320 --face=false

Throws following file not found error

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
models/pose/body_21a/pose_deploy.prototxt
models/pose/body_21a/pose_iter_XXXXXX.caffemodel
unable to read file
models/pose/body_21a/pose_iter_XXXXXX.caffemodel

Error:
Caffe trained model file not found: models/pose/body_21a/pose_iter_XXXXXX.caffemodel.
Possible causes:
    1. Not downloading the OpenPose trained models.
    2. Not running OpenPose from the same directory where the `model` folder is located.
    3. Using paths with spaces.

Coming from:
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/net/netCaffe.cpp:ImplNetCaffe():61
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/net/netCaffe.cpp:ImplNetCaffe():94
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorCaffe.cpp:addCaffeNetOnThread():108
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorCaffe.cpp:netInitializationOnThread():198
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorNet.cpp:initializationOnThread():98
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractor.cpp:initializationOnThread():34
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/pose/wPoseExtractor.hpp:initializationOnThread():57
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/thread/worker.hpp:initializationOnThreadNoException():77

Error occurred on a thread. OpenPose closed all its threads and then propagated the error to the main thread. Error description:

Caffe trained model file not found: models/pose/body_21a/pose_iter_XXXXXX.caffemodel.
Possible causes:
    1. Not downloading the OpenPose trained models.
    2. Not running OpenPose from the same directory where the `model` folder is located.
    3. Using paths with spaces.

Coming from:
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/net/netCaffe.cpp:ImplNetCaffe():61
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/net/netCaffe.cpp:ImplNetCaffe():94
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorCaffe.cpp:addCaffeNetOnThread():108
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorCaffe.cpp:netInitializationOnThread():198
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractorNet.cpp:initializationOnThread():98
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/pose/poseExtractor.cpp:initializationOnThread():34
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/pose/wPoseExtractor.hpp:initializationOnThread():57
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/thread/worker.hpp:initializationOnThreadNoException():77
- [All threads closed and control returned to main thread]
- /home/akunchala/Documents/PhDStuff/STAF/src/openpose/utilities/errorAndLog.cpp:checkWorkerErrors():280
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/thread/threadManager.hpp:stop():243
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/thread/threadManager.hpp:exec():202
- /home/akunchala/Documents/PhDStuff/STAF/include/openpose/wrapper/wrapper.hpp:exec():444

But if I use --tracking 1 then it is complaining about CUDNN_STATUS_NOT_INITIALIZED

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
RUNNING PoseExtractorCaffeStaf::PoseExtractorCaffeStaf
models/pose/body_21a_video/pose_deploy.prototxt
models/pose/body_21a_video/pose_iter_264000.caffemodel
F0824 08:13:28.170768 999345 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0)  CUDNN_STATUS_NOT_INITIALIZED
*** Check failure stack trace: ***
    @     0x7f527f3531c3  google::LogMessage::Fail()
    @     0x7f527f35825b  google::LogMessage::SendToLog()
    @     0x7f527f352ebf  google::LogMessage::Flush()
    @     0x7f527f3536ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f527ef9bf05  caffe::CuDNNConvolutionLayer<>::LayerSetUp()
    @     0x7f527f07e0bd  caffe::Net<>::Init()
    @     0x7f527f0804c0  caffe::Net<>::Net()
    @     0x7f527fa1127c  op::NetCaffe::initializationOnThread()
    @     0x7f527fa32643  op::PoseExtractorCaffeStaf::addCaffeNetOnThread()
    @     0x7f527fa33908  op::PoseExtractorCaffeStaf::netInitializationOnThread()
    @     0x7f527fa428c5  op::PoseExtractorNet::initializationOnThread()
    @     0x7f527fa28d87  op::PoseExtractor::initializationOnThread()
    @     0x7f527fa24227  op::WPoseExtractor<>::initializationOnThread()
    @     0x7f527fa72847  op::Worker<>::initializationOnThreadNoException()
    @     0x7f527fa72998  op::SubThread<>::initializationOnThread()
    @     0x7f527fa741c0  op::Thread<>::initializationOnThread()
    @     0x7f527fa7781c  op::Thread<>::threadFunction()
    @     0x7f527f692de4  (unknown)
    @     0x7f527f393609  start_thread
    @     0x7f527f4d1293  clone
Aborted (core dumped)

[mkocabas]

Yes, also we strictly need the tracking output from STAF since VIBE works on cropped images of a tracked person.

[anilkunchalaece]

Hi @mkocabas thanks for clarifying. Can we run VIBE with BODY_25 or COCO models with tracking option ?

Note : For anyone come across CUDNN_STATUS_NOT_INITIALIZED , In my case it is probably due to not enough GPU memory.

[mkocabas]

That is not possible at the moment. You can see this issue for STAF's authors response: https://github.com/soulslicer/STAF/issues/8