search

ZhengyiLuo / PHC

Official Implementation of the ICCV 2023 paper: Perpetual Humanoid Control for Real-time Simulated Avatars
https://zhengyiluo.github.io/PHC/
Other
488 stars 43 forks source link

Is this phenomenon normal? It feels like the robot's movements are not smooth. #87

Closed dbdxnuliba closed 1 week ago

dbdxnuliba commented 1 week ago

Hello, @ZhengyiLuo ,@kangnil

when I use PHC+ PHC+: keypoint model, can getup from the ground and walk back) | Best model for video/language model

I run the folloing command in the terminal:


python phc/run_hydra.py learning=im_mcp_big learning.params.network.ending_act=False exp_name=phc_comp_kp_2 env.obs_v=7 env=env_im_getup_mcp robot=smpl_humanoid robot.real_weight_porpotion_boxes=False env.motion_file=sample_data/amass_isaac_standing_upright_slim.pkl env.models=['output/HumanoidIm/phc_kp_2/Humanoid.pth'] env.num_prim=3 env.num_envs=1  headless=False epoch=-1 test=True

and the gif is as following: Is this phenomenon normal? It feels like the robot's movements are not smooth

test1

My pc is 4070ti image

and pip list:

(isaac) rob@rob:~/rl/PHC$ pip list
Package                       Version     Editable project location
----------------------------- ----------- -------------------------------------------------------
absl-py                       2.1.0
actionlib                     1.14.0
addict                        2.4.0
aiohappyeyeballs              2.4.3
aiohttp                       3.10.10
aiosignal                     1.3.1
angles                        1.9.13
antlr4-python3-runtime        4.9.3
asttokens                     2.4.1
async-timeout                 4.0.3
attrs                         24.2.0
autograd                      1.7.0
backcall                      0.2.0
beautifulsoup4                4.12.3
blinker                       1.8.2
bondpy                        1.8.6
Brotli                        1.0.9
cachetools                    5.5.0
camera-calibration            1.17.0
camera-calibration-parsers    1.12.0
catkin                        0.8.10
cchardet                      2.1.7
certifi                       2024.8.30
cffi                          1.17.1
chardet                       5.2.0
charset-normalizer            3.3.2
chumpy                        0.70
click                         8.1.7
cloudpickle                   3.1.0
comm                          0.2.2
ConfigArgParse                1.7
configer                      1.3.1
configparser                  7.1.0
contourpy                     1.1.1
controller-manager            0.20.0
controller-manager-msgs       0.20.0
cv-bridge                     1.16.2
cvxopt                        1.3.2
cycler                        0.12.1
Cython                        3.0.11
dash                          2.18.2
dash-core-components          2.0.0
dash-html-components          2.0.0
dash-table                    5.0.0
decorator                     5.1.1
diagnostic-analysis           1.11.0
diagnostic-common-diagnostics 1.11.0
diagnostic-updater            1.11.0
docker-pycreds                0.4.0
dynamic-reconfigure           1.7.3
easydict                      1.13
etils                         1.3.0
executing                     2.1.0
Farama-Notifications          0.0.4
fasteners                     0.19
fastjsonschema                2.20.0
filelock                      3.13.1
Flask                         3.0.3
fonttools                     4.54.1
freetype-py                   2.5.1
frozenlist                    1.5.0
fsspec                        2024.10.0
gazebo_plugins                2.9.2
gazebo_ros                    2.9.2
gdown                         5.2.0
gencpp                        0.7.0
geneus                        3.0.0
genlisp                       0.4.18
genmsg                        0.6.0
gennodejs                     2.0.2
genpy                         0.6.15
gitdb                         4.0.11
GitPython                     3.1.43
glfw                          2.7.0
gmpy2                         2.1.2
google-auth                   2.35.0
google-auth-oauthlib          1.0.0
grpcio                        1.64.0
gym                           0.26.2
gym-notices                   0.0.8
gymnasium                     1.0.0
human_body_prior              0.8.5.0
hydra-core                    1.3.2
idna                          3.7
image-geometry                1.16.2
imageio                       2.35.1
imageio-ffmpeg                0.5.1
importlib_metadata            8.5.0
importlib-resources           3.0.0
interactive-markers           1.12.0
ipdb                          0.13.13
ipython                       8.12.3
ipywidgets                    8.1.5
isaacgym                      1.0rc4      /home/rob/rl/IsaacGym_Preview_4_Package/isaacgym/python
itsdangerous                  2.2.0
jedi                          0.19.1
Jinja2                        3.1.4
joblib                        1.4.2
joint-state-publisher         1.15.1
joint-state-publisher-gui     1.15.1
jsonschema                    4.23.0
jsonschema-specifications     2023.12.1
jupyter_core                  5.7.2
jupyterlab_widgets            3.0.13
kiwisolver                    1.4.7
laser_geometry                1.6.7
lazy_loader                   0.4
lightning-utilities           0.11.8
lxml                          5.3.0
Markdown                      3.7
MarkupSafe                    2.1.3
matplotlib                    3.6.3
matplotlib-inline             0.1.7
mediapy                       1.2.2
message-filters               1.16.0
mkl-fft                       1.3.8
mkl-random                    1.2.4
mkl-service                   2.4.0
moveit-commander              1.1.13
moveit-core                   1.1.13
moveit-python                 0.4.5
moveit-ros-planning-interface 1.1.13
moveit-ros-visualization      1.1.13
moveit-task-constructor-core  0.1.3
mpmath                        1.3.0
msgpack                       1.1.0
mujoco                        3.2.3
mujoco-py                     2.1.2.14
multidict                     6.1.0
nbformat                      5.10.4
nest-asyncio                  1.6.0
networkx                      3.1
ninja                         1.11.1.1
numpy                         1.22.4
numpy-stl                     3.1.2
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
oauthlib                      3.2.2
omegaconf                     2.3.0
open3d                        0.18.0
opencv-python                 4.6.0.66
packaging                     24.1
pandas                        2.0.3
parso                         0.8.4
patchelf                      0.17.2.1
pexpect                       4.9.0
pickleshare                   0.7.5
pillow                        10.4.0
pip                           24.2
pkgutil_resolve_name          1.3.10
platformdirs                  4.3.6
plotly                        5.24.1
prompt_toolkit                3.0.48
propcache                     0.2.0
protobuf                      5.28.3
psutil                        6.1.0
ptyprocess                    0.7.0
pure_eval                     0.2.3
pyasn1                        0.6.1
pyasn1_modules                0.4.1
pycparser                     2.22
pyglet                        2.0.18
Pygments                      2.18.0
PyOpenGL                      3.1.7
pyparsing                     3.1.4
pyquaternion                  0.9.9
pyrender                      0.1.45
PySocks                       1.7.1
python-dateutil               2.9.0.post0
python-qt-binding             0.4.4
python-utils                  3.8.2
pytorch-lightning             2.4.0
pytz                          2024.2
PyVirtualDisplay              3.0
PyWavelets                    1.4.1
PyYAML                        6.0.2
qt-dotgraph                   0.4.2
qt-gui                        0.4.2
qt-gui-cpp                    0.4.2
qt-gui-py-common              0.4.2
ray                           2.10.0
referencing                   0.35.1
requests                      2.32.3
requests-oauthlib             2.0.0
resource_retriever            1.12.7
retrying                      1.3.4
rl-games                      1.1.4
rosbag                        1.16.0
rosboost-cfg                  1.15.8
rosclean                      1.15.8
roscreate                     1.15.8
rosgraph                      1.16.0
roslaunch                     1.16.0
roslib                        1.15.8
roslint                       0.12.0
roslz4                        1.16.0
rosmake                       1.15.8
rosmaster                     1.16.0
rosmsg                        1.16.0
rosnode                       1.16.0
rosparam                      1.16.0
rospy                         1.16.0
rosservice                    1.16.0
rostest                       1.16.0
rostopic                      1.16.0
rosunit                       1.15.8
roswtf                        1.16.0
rpds-py                       0.20.1
rqt_action                    0.4.9
rqt_bag                       0.5.1
rqt_bag_plugins               0.5.1
rqt-console                   0.4.12
rqt_dep                       0.4.12
rqt_graph                     0.4.14
rqt_gui                       0.5.3
rqt_gui_py                    0.5.3
rqt-image-view                0.4.17
rqt_launch                    0.4.9
rqt-logger-level              0.4.12
rqt-moveit                    0.5.11
rqt_msg                       0.4.10
rqt_nav_view                  0.5.7
rqt_plot                      0.4.13
rqt_pose_view                 0.5.11
rqt_publisher                 0.4.10
rqt_py_common                 0.5.3
rqt_py_console                0.4.10
rqt-reconfigure               0.5.5
rqt-robot-dashboard           0.5.8
rqt-robot-monitor             0.5.15
rqt_robot_steering            0.5.12
rqt-runtime-monitor           0.5.10
rqt-rviz                      0.7.0
rqt_service_caller            0.4.10
rqt_shell                     0.4.11
rqt_srv                       0.4.9
rqt-tf-tree                   0.6.4
rqt_top                       0.4.10
rqt_topic                     0.4.13
rqt_web                       0.4.10
rsa                           4.9
rviz                          1.14.20
scikit-image                  0.21.0
scikit-learn                  1.3.2
scipy                         1.10.1
sensor-msgs                   1.13.1
sentry-sdk                    2.18.0
setproctitle                  1.3.3
setuptools                    75.1.0
six                           1.16.0
smach                         2.5.2
smach-ros                     2.5.2
smclib                        1.8.6
smmap                         5.0.1
smpl_sim                      0.0.1
smplx                         0.1.28
soupsieve                     2.6
srdfdom                       0.6.4
stack-data                    0.6.3
sympy                         1.13.2
tenacity                      9.0.0
tensorboard                   2.14.0
tensorboard-data-server       0.7.2
tensorboardX                  2.6.2.2
termcolor                     2.4.0
tf                            1.13.2
tf-conversions                1.13.2
tf2-geometry-msgs             0.7.7
tf2-kdl                       0.7.7
tf2-py                        0.7.7
tf2-ros                       0.7.7
threadpoolctl                 3.5.0
tifffile                      2023.7.10
tomli                         2.0.2
topic-tools                   1.16.0
torch                         1.13.1
torchaudio                    2.4.1
torchgeometry                 0.1.2
torchmetrics                  1.5.1
torchvision                   0.14.1
tqdm                          4.66.6
trac_ik_python                1.6.6
traitlets                     5.14.3
transforms3d                  0.4.2
trimesh                       4.5.2
typing_extensions             4.11.0
tzdata                        2024.2
urdfdom-py                    0.4.6
urllib3                       2.2.3
vtk                           9.3.1
wandb                         0.18.5
wcwidth                       0.2.13
Werkzeug                      3.0.6
wheel                         0.44.0
widgetsnbextension            4.0.13
xacro                         1.14.18
yarl                          1.15.2
zipp                          3.20.2
ZhengyiLuo commented 1 week ago

That looks like normal standing motion to me. The "jump" you see is the episode reset. When the episode is reset, the humanoid will be re-initialized, which can cause a jump (it's like you are just put in a new environment and need a little time to reestablish balance). Afterwards imitating the standing motion looks pretty stable to me (almost not moving)?

dbdxnuliba commented 1 week ago

many thanks