ImportError: No module named msg when deploy on a real robot

[bach05]

Hi, I'm trying to use the rt_gene ros package on the TIAGo robot from PAL. The system works on my PC, but when I deploy (namely I send the packages to the robot with the official script) rt_gene, I got this errors:

ROS_MASTER_URI=http://control:11311

process[gaze/gaze_extract_landmarks_new-1]: started with pid [13436]
process[gaze/gaze_estimate_gaze_twoeyes-2]: started with pid [13437]
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/rt_gene/estimate_gaze.py", line 22, in <module>
    from rt_gene.msg import MSG_Gaze, MSG_GazeList
ImportError: No module named msg
[gaze/gaze_estimate_gaze_twoeyes-2] process has died [pid 13437, exit code 1, cmd /home/pal/deployed_ws/lib/rt_gene/estimate_gaze.py __name:=gaze_estimate_gaze_twoeyes __log:=/home/pal/.ros/log/f4a72a00-f03d-11eb-b93c-04d4c48fb41c/gaze-gaze_estimate_gaze_twoeyes-2.log].
log file: /home/pal/.ros/log/f4a72a00-f03d-11eb-b93c-04d4c48fb41c/gaze-gaze_estimate_gaze_twoeyes-2*.log
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/rt_gene/extract_landmarks_node.py", line 31, in <module>
    from rt_gene.msg import MSG_SubjectImagesList
ImportError: No module named msg
[gaze/gaze_extract_landmarks_new-1] process has died [pid 13436, exit code 1, cmd /home/pal/deployed_ws/lib/rt_gene/extract_landmarks_node.py /camera_info:=/xtion/rgb/camera_info /image:=/xtion/rgb/image_rect_color __name:=gaze_extract_landmarks_new __log:=/home/pal/.ros/log/f4a72a00-f03d-11eb-b93c-04d4c48fb41c/gaze-gaze_extract_landmarks_new-1.log].
log file: /home/pal/.ros/log/f4a72a00-f03d-11eb-b93c-04d4c48fb41c/gaze-gaze_extract_landmarks_new-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done

I also try to create a simple ros package with some messages and it works on the robot. I also checked if the msg module was present and it is the case. In fact, I have a folder ~/deployed_ws/lib/python2.7/dist-packages/rt_gene/msg which contains the .py and .pyc message files.

I really don't know how to solve the problem. Thank you in advance for your help.

[Tobias-Fischer]

Have you tried doing this? the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'

[Tobias-Fischer]

Actually, the rosdep stuff should be independent of the problem you are facing.

Which ROS version are you using?

Can you please provide the output of python -c "import rt_gene; print(rt_gene.__file__)" and python -c "import rt_gene.msg; print(rt_gene.msg.__file__)"?

[bach05]

Thank you for your answer. Yes of course.

I'm using ROS Melodic. This is the output:

pal@tiago-87c:~$ python -c "import rt_gene; print(rt_gene.__file__)"
/home/pal/deployed_ws/lib/python2.7/dist-packages/rt_gene/__init__.pyc
pal@tiago-87c:~$ python -c "import rt_gene.msg; print(rt_gene.msg.__file__)"
/home/pal/deployed_ws/lib/python2.7/dist-packages/rt_gene/msg/__init__.pyc

Tell me if you need something more.

[Tobias-Fischer]

So that's very weird, I'm not sure what's going on. It can clearly find rt_gene.msg there, but not in the extract landmarks node.

@ahmed-alhindawi - do you have any ideas?

[bach05]

Also rosmsg list is able to show the messages.

[Tobias-Fischer]

What if you try python -c "from rt_gene.msg import MSG_SubjectImagesList"? Does that work without issues?

[bach05]

I tried something similar. I don't remember exactly which message now, but I remenber clearly that there was no issues.

In this moment I have no access to the robot, but I think it will be the same with python -c "from rt_gene.msg import MSG_SubjectImagesList".

[ahmed-alhindawi]

Sorry for being late to this issue - could this be a PYTHONPATH problem? Perhaps try printing the PYTHONPATH on your PC and then on the robot to see if there are issues finding the messages? Otherwise, I'm afraid I'm out of ideas...

[bach05]

Sorry for the late response, I'm now with the TIAGo. These are the PYTHONPATH prints.

ROBOT:

python 
Python 2.7.17 (default, Nov  7 2019, 10:07:09) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/home/pal/deployed_ws/lib/python2.7/dist-packages', '/opt/pal/ferrum/lib/python2.7/dist-packages', '/opt/ros/melodic/lib/python2.7/dist-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/pal/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk3']
>>> import os
>>> os.environ['PYTHONPATH'].split(os.pathsep)
['/home/pal/deployed_ws/lib/python2.7/dist-packages', '/opt/pal/ferrum/lib/python2.7/dist-packages', '/opt/ros/melodic/lib/python2.7/dist-packages']

PC:

python
Python 2.7.17 (default, Nov  7 2019, 10:07:09) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/root/bacchin_ws/devel/lib/python2.7/dist-packages', '/root/catkin_ws/devel/lib/python2.7/dist-packages', '/opt/pal/ferrum/lib/python2.7/dist-packages', '/opt/ros/melodic/lib/python2.7/dist-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk3']
>>> import os
>>> os.environ['PYTHONPATH'].split(os.pathsep)
['/root/bacchin_ws/devel/lib/python2.7/dist-packages', '/root/catkin_ws/devel/lib/python2.7/dist-packages', '/opt/pal/ferrum/lib/python2.7/dist-packages', '/opt/ros/melodic/lib/python2.7/dist-packages']

Are there some problems?

[Tobias-Fischer]

Could you please list all files in /home/pal/deployed_ws/lib/python2.7/dist-packages/rt_gene/?

[bach05]

Of course:

~/deployed_ws/lib/python2.7/dist-packages/rt_gene$ ls
cfg                                 gaze_tools_standalone.pyc
download_tools.py                   __init__.py
download_tools.pyc                  __init__.pyc
estimate_gaze_base.py               kalman_stabilizer.py
estimate_gaze_base.pyc              kalman_stabilizer.pyc
estimate_gaze_pytorch.py            msg
estimate_gaze_pytorch.pyc           ros_tools.py
estimate_gaze_tensorflow.py         ros_tools.pyc
estimate_gaze_tensorflow.pyc        subject_ros_bridge.py
extract_landmarks_method_base.py    subject_ros_bridge.pyc
extract_landmarks_method_base.pyc   tracker_face_encoding.py
gaze_estimation_models_pytorch.py   tracker_face_encoding.pyc
gaze_estimation_models_pytorch.pyc  tracker_generic.py
gaze_tools.py                       tracker_generic.pyc
gaze_tools.pyc                      tracker_sequential.py
gaze_tools_standalone.py            tracker_sequential.pyc

[Tobias-Fischer]

And the ones in abc/rt_gene/msg?

[Tobias-Fischer]

Sorry - I meant ~/deployed_ws/lib/python2.7/dist-packages/rt_gene/msg

[bach05]

Sorry, I feel a little dumb this morning. Now I got it:

~/deployed_ws/lib/python2.7/dist-packages/rt_gene/msg$ ls
__init__.py           _MSG_HeadposeList.pyc
__init__.pyc          _MSG_Headpose.py
_MSG_BlinkList.py     _MSG_Headpose.pyc
_MSG_BlinkList.pyc    _MSG_LandmarksList.py
_MSG_Blink.py         _MSG_LandmarksList.pyc
_MSG_Blink.pyc        _MSG_Landmarks.py
_MSG_GazeList.py      _MSG_Landmarks.pyc
_MSG_GazeList.pyc     _MSG_SubjectImagesList.py
_MSG_Gaze.py          _MSG_SubjectImagesList.pyc
_MSG_Gaze.pyc         _MSG_SubjectImages.py

[Tobias-Fischer]

Hi @bach05 - that all looks okay to me, and seeing that you can do import rt_gene.msg; print(rt_gene.msg.__file__) without any issues (which is exactly the same as what happens in the code) I am really out of ideas, sorry. One last attempt would be to try whether python -c "from rt_gene.msg import MSG_Gaze, MSG_GazeList" works?

[Tobias-Fischer]

And maybe also try rosmsg show rt_gene/MSG_Gaze and rosmsg package rt_gene

[bach05]

Here I am. This is the output of the commands you asked me:

pal@tiago-87c:~$ python -c "from rt_gene.msg import MSG_Gaze, MSG_GazeList"
pal@tiago-87c:~$ rosmsg show rt_gene/MSG_Gaze
string subject_id
float64 phi
float64 theta

pal@tiago-87c:~$ rosmsg package rt_gene
rt_gene/MSG_Blink
rt_gene/MSG_BlinkList
rt_gene/MSG_Gaze
rt_gene/MSG_GazeList
rt_gene/MSG_Headpose
rt_gene/MSG_HeadposeList
rt_gene/MSG_Landmarks
rt_gene/MSG_LandmarksList
rt_gene/MSG_SubjectImages
rt_gene/MSG_SubjectImagesList
pal@tiago-87c:~$ 

[bach05]

I tried also to write this simple node in a different package:

#!/usr/bin/env python

import rospy
from msg_py.msg import prova #my custom messagge just to try
from rt_gene.msg import MSG_Gaze, MSG_GazeList
from rt_gene.msg import MSG_SubjectImagesList

def prova1():

    pub = rospy.Publisher('/prova',prova,queue_size=10)
    rospy.init_node('prova1',anonymous=True)

    rate = rospy.Rate(10)
    while not rospy.is_shutdown():
        rate.sleep()

if __name__ == '__main__':
    try:
        prova1()
    except rospy.ROSInterruptException:
        pass

And there are no errors. It sounds really weird. Maybe is there some problems in the rt_gene package organization? I'm going to try to get out some nodes from the original package.

Maybe it is related to the installation section of CMakeLists, since I'm not cloning directly the library.

[bach05]

UPDATE: I tried to get out the estimate_gaze.py in a different package, just to try:

rosrun msg_py estimate_gaze.py 
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/msg_py/estimate_gaze.py", line 182, in <module>
    [os.path.join(rospkg.RosPack().get_path("rt_gene"), model_file) for model_file in rospy.get_param("~model_files")])
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/client.py", line 465, in get_param
    return _param_server[param_name] #MasterProxy does all the magic for us
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/msproxy.py", line 123, in __getitem__
    raise KeyError(key)
KeyError: '~model_files'

It seems the issue with messages is not present, isn't it? There are other errors, I guess because of missing files from the original package. It should be definitely a problem with rt_gene package organization/installation.

UPDATE 2: if I try to use roslaunch in the new package, I get the same errors with the import rt_gene.msg. I really don't understand where is the problem...

[Tobias-Fischer]

Hmmm, that is very weird indeed, and I am not sure how to debug this further.

I guess you already tried cleaning the workspace and rebuilding it? Or putting the rt_gene package in a separate catkin workspace?

Worst case, (and I acknowledge that this would be far from optimal), but could you run the two rt-gene nodes via rosrun separately from your other launch files?

[Tobias-Fischer]

Did you end up having any luck @bach05?

[bach05]

Excuse me, I haven't answered yet because I have no access to the robot in this period (due to holidays).

However, I may have an idea. Since we have also an NVIDIA Jetson TX2 on board, do you think is it possible to run rt-gene there? By chance, do you have a docker available for your application?

[Tobias-Fischer]

I don't see why it would not work. However, the installation could be a bit complex due to the different architecture of the Jetson. You would need to use particular versions of PyTorch, OpenCV, ROS etc; it would involve some work.

There is no Docker, sorry.

[bach05]

Hi, I tried some further tests. I tried to run the nodes with rosrun:

rosrun rt_gene2 estimate_gaze.py _model_files:=['model_nets/Model_allsubjects1.h5'] _tf_prefix:=gaze _gaze_backend:=pytorch device_id_gazeestimation:=cpu:0 _visualise_eyepose:=True
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/rt_gene2/estimate_gaze.py", line 22, in <module>
    from rt_gene.msg import MSG_Gaze, MSG_GazeList
ImportError: No module named msg

rosrun rt_gene extract_landmarks_node.py _device_id_facedetection:=True _use_face_encoding_tracker:=0.8 _tf_prefix:=gaze _visualise_headpose:=True /camera_info:=/xtion/rgb/camera_info /image:=/xtion/rgb/image_rect_color
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/rt_gene/extract_landmarks_node.py", line 31, in <module>
    from rt_gene.msg import MSG_SubjectImagesList
ImportError: No module named msg

rosrun rt_gene2 extract_landmarks_node.py _device_id_facedetection:=True _use_face_encoding_tracker:=0.8 _tf_prefix:=gaze _visualise_headpose:=True /camera_info:=/xtion/rgb/camera_info /image:=/xtion/rgb/image_rect_color
the rosdep view is empty: call 'sudo rosdep init' and 'rosdep update'
Traceback (most recent call last):
  File "/home/pal/deployed_ws/lib/rt_gene2/extract_landmarks_node.py", line 31, in <module>
    from rt_gene.msg import MSG_SubjectImagesList
ImportError: No module named msg

Note that rt_gene2 is the package I have obtained by getting out the files from the original one, except for messages because if you remember it seems that an external package was able to find the messages. But no way.

I could suppose there is a problem in the organization of the package, but I can't see it.

In every case, thank you for your help. If I find a solution, I will write it here!

[Tobias-Fischer]

So I had another think about your issues. I think they are because of the PAL script that copies over the package to the robot - the structure is weird. That's why it works for basically anyone else, including yourself on your desktop machine.

To get around it, I suggest to create a separate package rt_gene_msgs which only contains the messages. So move all the messages + message generation / runtime things to this separate package, and update all references from rt_gene.msg to rt_gene_msgs.msg. This should then work. Are you okay in doing that? I'm pretty sure it'll solve your problems.

[bach05]

This could be a good idea! I have temporarily solved the problem using an external laptop in place of the onboard PC. The rate is a little slower, but it works fine. Since I have some deadlines, I will try your suggestion as soon as possible. Sorry if it will take some time.

Now I am dealing with another problem: since people must wear masks, the estimation is not so reliable. I think because the detector is not able to retrieve all the key points. You use eye, mouth, and nose key points if I remember well. Maybe can we use different landmarks like the contours of the face (as dlib 68 landmarks)?

Do a GPU improve the reliability of just the speed?

[Tobias-Fischer]

Hi, a long time ago we used dlib. We found that it doesn't work as well as the current solution; masks are a different story though and I guess a research problem on its own. Swapping out the landmark detection code shouldn't be too hard, but I'm afraid that due to time constraints I won't be able to help out with that. If you look through the git history you might be able to see at some point where we swapped them.

[Tobias-Fischer]

A GPU helps purely in faster inference, not improved accuracy. However indirectly fast inference might help as otherwise some data points (images) might not get processed which leads to issues down the track with regards to the Kalman filter etc

[bach05]

Thank you very much for your suggestions! You have been very kind. I will try to have a look inside the code or at least we can try with transparent masl that may be the fastest solution.

I will also try to deploy the system on the robot as soon as I find a free moment.

[Tobias-Fischer]

Great! I'm closing this issue for now - feel free to re-open if you have more questions.