search

YinghaoHuang91 / MuVS

This is the code for MuVS (Multi-view SMPLify)
MIT License
112 stars 23 forks source link

Strange Pose fitting results when using OpenPose J2D input #16

Closed wine3603 closed 4 years ago

wine3603 commented 4 years ago

Hi man, I changed your Deepcut joint input to OpenPose 18 joints. I removed the Head_top joint of DeepCut and mapped the nose_vertex_ID to openpose nose joint. I only changed the codes in util.py and smpl_batch.py didn't touch frame_fit.py. The result of frame_006 looks strange: f6_wrong00

the right one use ur original DeepCut input is: f6_deepcut00

The print output is:

./gen_run_jobs_1.sh
0
../Data/HEVA_Validate/S1_Box_1_C1/Image/frame0006.png
Tensor("strided_slice_136:0", shape=(1, 13, 3), dtype=float32)
WARNING:tensorflow:From frame_fit.py:92: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-09-22 14:22:43.737382: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-22 14:22:43.801989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:05:00.0
2020-09-22 14:22:43.803034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:06:00.0
2020-09-22 14:22:43.804058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:09:00.0
2020-09-22 14:22:43.804240: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.804335: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.804423: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.804510: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.804599: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.804686: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2020-09-22 14:22:43.807522: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-22 14:22:43.807549: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...
2020-09-22 14:22:43.807838: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-09-22 14:22:44.152283: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557cfd57c100 executing computations on platform CUDA. Devices:
2020-09-22 14:22:44.152340: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-09-22 14:22:44.152359: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-09-22 14:22:44.152373: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-09-22 14:22:44.175759: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3498000000 Hz
2020-09-22 14:22:44.176515: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557cfc112f80 executing computations on platform Host. Devices:
2020-09-22 14:22:44.176566: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2020-09-22 14:22:44.176735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-22 14:22:44.176770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]
WARNING:tensorflow:From frame_fit.py:93: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

2020-09-22 14:22:44.234960: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.

RUNNING THE L-BFGS-B CODE

   * * *

Machine precision = 2.220D-16
N = 6 M = 10

This problem is unconstrained.

At X0 0 variables are exactly at the bounds

Figure_1-7

At iterate 0 f= 3.32008D+04 |proj g|= 5.35095D+04

At iterate 1 f= 5.12031D+03 |proj g|= 7.11054D+03

At iterate 2 f= 3.77815D+03 |proj g|= 5.47154D+03

At iterate 3 f= 2.42080D+03 |proj g|= 3.95927D+03

At iterate 4 f= 1.36007D+03 |proj g|= 3.27027D+03

At iterate 5 f= 7.52664D+02 |proj g|= 9.48627D+02

At iterate 6 f= 6.89472D+02 |proj g|= 1.83354D+02

At iterate 7 f= 6.84448D+02 |proj g|= 1.01980D+02

At iterate 8 f= 6.76808D+02 |proj g|= 1.95080D+02

At iterate 9 f= 6.55143D+02 |proj g|= 1.02498D+03

At iterate 10 f= 5.22447D+02 |proj g|= 1.65344D+03

At iterate 11 f= 3.32857D+02 |proj g|= 2.76960D+03

At iterate 12 f= 2.82925D+02 |proj g|= 2.34534D+03

At iterate 13 f= 1.92021D+02 |proj g|= 5.68519D+02

At iterate 14 f= 1.68120D+02 |proj g|= 3.90624D+02

At iterate 15 f= 1.62494D+02 |proj g|= 2.04222D+02

At iterate 16 f= 1.56634D+02 |proj g|= 1.65284D+02

At iterate 17 f= 1.49091D+02 |proj g|= 1.42732D+02

At iterate 18 f= 1.45896D+02 |proj g|= 2.77377D+02

At iterate 19 f= 1.45014D+02 |proj g|= 4.93933D+01

Figure_1-11

At iterate 20 f= 1.44913D+02 |proj g|= 6.80469D+00

       * * *

Tit = total number of iterations
Tnf = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip = number of BFGS updates skipped
Nact = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F = final function value

       * * *

N Tit Tnf Tnint Skip Nact Projg F
6 20 25 1 0 0 6.805D+00 1.449D+02
F = 144.91284179687500

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH

Cauchy time 0.000E+00 seconds.
Subspace minimization time 0.000E+00 seconds.
Line search time 0.000E+00 seconds.

Total User time 0.000E+00 seconds.

RUNNING THE L-BFGS-B CODE

       * * *

Machine precision = 2.220D-16
N = 85 M = 10
This problem is unconstrained.

At X0 0 variables are exactly at the bounds

At iterate 0 f= 5.09612D+04 |proj g|= 2.31368D+04

Figure_1-8

At iterate 1 f= 4.88808D+04 |proj g|= 2.98825D+04

Figure_1-9

At iterate 2 f= 4.70862D+04 |proj g|= 2.49123D+04

At iterate 3 f= 4.32866D+04 |proj g|= 2.85334D+04

At iterate 4 f= 4.06109D+04 |proj g|= 2.34374D+04

At iterate 5 f= 3.98855D+04 |proj g|= 4.84759D+03

At iterate 6 f= 3.95298D+04 |proj g|= 4.49670D+03

Figure_1-10

At iterate 7 f= 3.85606D+04 |proj g|= 1.30690D+04

At iterate 8 f= 3.79381D+04 |proj g|= 8.20185D+03

At iterate 9 f= 3.73606D+04 |proj g|= 2.97926D+03

At iterate 10 f= 3.68244D+04 |proj g|= 7.99022D+03

At iterate 11 f= 3.63427D+04 |proj g|= 6.89147D+03

At iterate 12 f= 3.58900D+04 |proj g|= 5.91139D+03

At iterate 13 f= 3.56545D+04 |proj g|= 9.22863D+03

At iterate 14 f= 3.54833D+04 |proj g|= 1.25315D+04

At iterate 15 f= 3.53099D+04 |proj g|= 3.23353D+03

At iterate 16 f= 3.51866D+04 |proj g|= 2.60039D+03

At iterate 17 f= 3.49947D+04 |proj g|= 7.10795D+03

At iterate 18 f= 3.47829D+04 |proj g|= 5.58012D+03

At iterate 19 f= 3.46971D+04 |proj g|= 9.31179D+03

At iterate 20 f= 3.45636D+04 |proj g|= 3.14364D+03

At iterate 21 f= 3.45150D+04 |proj g|= 8.36188D+02

Figure_1-12

At iterate 22 f= 3.44934D+04 |proj g|= 1.46735D+03

       * * *

Tit = total number of iterations
Tnf = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip = number of BFGS updates skipped
Nact = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F = final function value

       * * *

N Tit Tnf Tnint Skip Nact Projg F
85 22 25 1 0 0 1.467D+03 3.449D+04
F = 34493.414062500000

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH

Cauchy time 0.000E+00 seconds.
Subspace minimization time 0.000E+00 seconds.
Line search time 0.000E+00 seconds.

Total User time 0.000E+00 seconds.

We can see in the second optimizing round, the F value is big but almost not change, which finally results in converging.
The actor's left arm cannot be optimized until the last. Could you give me any advice about where is the problem?

Update: the problem seems like the 2nd optimization falls into the local minimum problems thus ends too early. In some frames, such as frame 30, It runs good, 65 iterations and I got the same result with deep-cut input since its optimization. Could you help me to figure out why the optimization always ends up with D+04, such a big f value?

wine3603 commented 4 years ago

One more point is about the joint weight. where did you set each joint weight? Since the openpose hip and nect joints are far away from SMPL's, I want to set their weights as 0, like SMPLifyx did: #

wine3603 commented 4 years ago

I fixed this problem. I'll open a PR of using openpose joints afterward.

iluvrachel commented 4 years ago

I fixed this problem. I'll open a PR of using openpose joints afterward.

Looking forward to your code!!

nicholas-kondal commented 3 years ago

Hi @wine3603, great job on using OpenPose inputs! I'm trying to do the same but having some trouble (unsure what exactly to modify in util.py and smpl_batch.py), so I was wondering if you made any progress on that PR?

nicholas-kondal commented 3 years ago

@wine3603 Did you use these SMPL joint IDs for the OpenPose 18 joints?

For the nose, eyes, and ears specifically, I believe that we would need to map them to SMPL vertices listed vertex_ids.py in the SMPLX code, correct?