How to extract data from demo.bag

[yuan8421]

Hi community,

1) I want to add another car to the simulation. 2) Before adding a driving planner I need to add cars besides the driving car. 3) I need to analysis demo.bag and made my new_demo.bag

I run this command: rostopic echo -b demo.bag -p /topic > data.txt

I got these message:

WARNING: For type [pb_msgs/Gps] stored md5sum [8fad5985ce947d3b6854fd093a59c429] has invalid message definition." WARNING: For type [pb_msgs/Imu] stored md5sum [bdef0ba51869607ed95736d41e80c1f5] has invalid message definition." WARNING: For type [pb_msgs/Chassis] stored md5sum [d6a21658031a6a4615858d76f8b5178e] has invalid message definition." WARNING: For type [pb_msgs/ControlCommand] stored md5sum [67f7ff8a4c675dc97a8c7ce6d6289943] has invalid message definition." WARNING: For type [pb_msgs/MonitorMessage] stored md5sum [ada339d65883d2037c88f67d8b43f957] has invalid message definition." WARNING: For type [pb_msgs/PadMessage] stored md5sum [35ef81c50bd776c0c53a0b3b8572bb6e] has invalid message definition." WARNING: For type [pb_msgs/LocalizationEstimate] stored md5sum [503c8e75900db180bc61534806a37cfb] has invalid message definition." WARNING: For type [pb_msgs/GnssStatus] stored md5sum [6ab9bfa7e56e2724f6d30280b731fef2] has invalid message definition." WARNING: For type [pb_msgs/ADCTrajectory] stored md5sum [97587fe9a5b2df2b61888d56c6fc697b] has invalid message definition."

Can anybody teach me how to extract the demo.bag?

Many thanks!

[yuan8421]

I find a python script in this path: apollo/modules/tools/dump_rosbag/dumpbag.py

I run this command: python dumpbag.py demo.bag demo/

However, I got the exact same message:

WARNING: For type [pb_msgs/Gps] stored md5sum [8fad5985ce947d3b6854fd093a59c429] has invalid message definition." WARNING: For type [pb_msgs/Imu] stored md5sum [bdef0ba51869607ed95736d41e80c1f5] has invalid message definition." WARNING: For type [pb_msgs/Chassis] stored md5sum [d6a21658031a6a4615858d76f8b5178e] has invalid message definition." WARNING: For type [pb_msgs/ControlCommand] stored md5sum [67f7ff8a4c675dc97a8c7ce6d6289943] has invalid message definition." WARNING: For type [pb_msgs/MonitorMessage] stored md5sum [ada339d65883d2037c88f67d8b43f957] has invalid message definition." WARNING: For type [pb_msgs/PadMessage] stored md5sum [35ef81c50bd776c0c53a0b3b8572bb6e] has invalid message definition." WARNING: For type [pb_msgs/LocalizationEstimate] stored md5sum [503c8e75900db180bc61534806a37cfb] has invalid message definition." WARNING: For type [pb_msgs/GnssStatus] stored md5sum [6ab9bfa7e56e2724f6d30280b731fef2] has invalid message definition." WARNING: For type [pb_msgs/ADCTrajectory] stored md5sum [97587fe9a5b2df2b61888d56c6fc697b] has invalid message definition."

[yuan8421]

I found this code in Module rosbag.bag: http://docs.ros.org/jade/api/rosbag/html/python/rosbag.bag-pysrc.html

def _get_message_type(info):
    message_type = _message_types.get(info.md5sum) 
    if message_type is None: 
        try: 
            message_type = genpy.dynamic.generate_dynamic(info.datatype, info.msg_def)[info.datatype] 
            if (message_type._md5sum != info.md5sum): 
                print('WARNING: For type [%s] stored md5sum [%s] does not match message definition [%s].\n  Try: "rosrun rosbag fix_msg_defs.py old_bag new_bag."'%(info.datatype, info.md5sum, message_type._md5sum), file=sys.stderr) 
        except genmsg.InvalidMsgSpec: 
            message_type = genpy.dynamic.generate_dynamic(info.datatype, "")[info.datatype] 
            print('WARNING: For type [%s] stored md5sum [%s] has invalid message definition."'%(info.datatype, info.md5sum), file=sys.stderr) 
        except genmsg.MsgGenerationException as ex: 
            raise ROSBagException('Error generating datatype %s: %s' % (info.datatype, str(ex))) 

        _message_types[info.md5sum] = message_type

    return message_type

This line print the warning message:

except genmsg.InvalidMsgSpec: 
            message_type = genpy.dynamic.generate_dynamic(info.datatype, "")[info.datatype] 
            print('WARNING: For type [%s] stored md5sum [%s] has invalid message definition."'%(info.datatype, info.md5sum), file=sys.stderr) 

[GoLancer]

Did you use the ros under /apollo/third_party/ros in docker？

[yuan8421]

@GoLancer Thank you for your hint! No, I didn't run ros command in apollo docker. I will try and tell you the result. Thanks!

[startcode]

@yuan8421 I have seen this before. The problem is that Python could not find the generated proto py files. Bazel actually created these files in bazel-genfiles, if you have successfully built apollo before. This problem can be address by two steps. In the apollo directory (where apollo.bash exists),

1. source the ros setup.bash by

   source bazel-apollo/external/ros/setup.bash

2. Set the PYTHONPATH variable.

   export PYTHONPATH=$(pwd)/bazel-genfiles:$PYTHONPATH

   Try this command to test if the problem is solved

   python modules/tools/dump_rosbag/dumpbag.py docs/demo_guide/demo.bag message_folder

   It will dump the docs/demo_guide/demo.bag into message_folder

[yuan8421]

Hi @startcode Thank you for the hints! However I followed your instruction, and I got this error message:

Traceback (most recent call last): File "modules/tools/dump_rosbag/dumpbag.py", line 72, in dump_bag(args.in_rosbag, args.out_dir, args.topic) File "modules/tools/dump_rosbag/dumpbag.py", line 43, in dump_bag for topic, msg, t in bag.read_messages(): File "/home/yuan8421/Work/git/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 2341, in read_messages yield self.seek_and_read_message_data_record((entry.chunk_pos, entry.offset), raw) File "/home/yuan8421/Work/git/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 2479, in seek_and_read_message_data_record msg_type = _get_message_type(connection_info) File "/home/yuan8421/Work/git/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 1572, in _get_message_type message_type = getattr(obj.msg, info.datatype.split('/')[1]) AttributeError: 'module' object has no attribute 'Gps'

BTW, I'm also interested in composing a rosbag too. Would you give me some hints, please?

Many thanks!

[yuan8421]

Hi @GoLancer Now I run the command in Docker, however still doesn't work. You can find the result in the previous reply. Thanks anyway!

[GoLancer]

@yuan8421 From your error message, it seems there is no proto py files in modules directory, can you post the output of the following commands:

$ ls /apollo/bazel-genfiles/modules
$ python -c "import modules"

[yuan8421]

Hi @GoLancer

Here is the message:

yuan8421@yuan:~/Work/git/apollo$ ls -lh bazel-genfiles/modules/

total 44K drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 canbus drwxr-xr-x 7 yuan8421 yuan8421 4.0K Jul 12 20:39 common drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 control drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 decision drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 dreamview drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 8 22:36 hmi -rw-r--r-- 1 yuan8421 yuan8421 0 Jul 8 17:45 init.py -rw-r--r-- 1 yuan8421 yuan8421 129 Jul 8 22:36 init.pyc drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 localization drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 perception drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 planning drwxr-xr-x 3 yuan8421 yuan8421 4.0K Jul 12 20:39 prediction

yuan8421@yuan:~/Work/git/apollo$ python -c "import modules"

Yes. I didn't find there is no proto there. What should I do next?

[startcode]

One difference I noticed is that your have an init.py file, but does not have a __init__.py file. Python requires a __init__.py file to recognize a package. These __init__.py files are generated by apollo.bash

find bazel-genfiles/* -type d -exec touch "{}/__init__.py" \;

You actually can re-run this command to generate the __init__.py files again.

Did you make any change in the build script?

[yuan8421]

Hi @startcode I solved the problem after following your instruction, thank you very much!

This is the solution

cd apollo/
docker ps
bash docker/scripts/dev_start.sh
bash docker/scripts/dev_into.sh
bash apollo.sh build
source bazel-apollo/external/ros/setup.bash 
export PYTHONPATH=$(pwd)/bazel-genfiles:$PYTHONPATH
mkdir yuan
python modules/tools/dump_rosbag/dumpbag.py docs/demo_guide/demo.bag yuan/

For who want to extract the rosbag developer, this is my way to extract. Some issue you should check:

1. Make sure you are work in Docker
2. Before you run python modules/tools/dump_rosbag/dumpbag.py docs/demo_guide/demo.bag yuan/, you should check if init.py exists in the directory bazel-genfiles/
3. Good luck!

Again, thank you! @startcode Thread close.

[fisher2020]

hi,i met the same question in apollo 1.5 docker environment ,even thouth i had tried your solutions,do you know how to solve this? thanks @yuan8421 @startcode

[yuan8421]

Hi @fisher2020 I clone the clean 1.5 apollo, and this is my console command line history Note: they change script from _modules/tools/dumprosbag/dumpbag.py to modules/tools/rosbag/dumpbag.py

bash docker/scripts/install_docker.sh docker ps bash docker/scripts/dev_start.sh bash docker/scripts/dev_into.sh bash apollo.sh build source bazel-apollo/external/ros/setup.bash export PYTHONPATH=$(pwd)/bazel-genfiles:$PYTHONPATH mkdir yuan python modules/tools/rosbag/dumpbag.py docs/demo_guide/demo.bag yuan/ cd yuan/ ls history

Everything works.

[fisher2020]

hi，@yuan8421 I clone the clean 1.5 apollo too，and do exactly what you show me，but met the error。 [ OK ] Build passed! [INFO] Took 493 seconds

root@in_dev_docker:/apollo# source bazel-apollo/external/ros/setup.bash root@in_dev_docker:/apollo# export PYTHONPATH=$(pwd)/bazel-genfiles:$PYTHONPATH root@in_dev_docker:/apollo# mkdir yuan root@in_dev_docker:/apollo# python modules/tools/rosbag/dumpbag.py docs/demo_guide/demo.bag yuan/ Traceback (most recent call last): File "modules/tools/rosbag/dumpbag.py", line 75, in dump_bag(args.in_rosbag, args.out_dir, args.topic) File "modules/tools/rosbag/dumpbag.py", line 43, in dump_bag for topic, msg, t in bag.read_messages(): File "/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 2341, in read_messages yield self.seek_and_read_message_data_record((entry.chunk_pos, entry.offset), raw) File "/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 2479, in seek_and_read_message_data_record msg_type = _get_message_type(connection_info) File "/apollo/bazel-apollo/external/ros/lib/python2.7/dist-packages/rosbag/bag.py", line 1572, in _get_message_type message_type = getattr(obj.msg, info.datatype.split('/')[1]) AttributeError: 'module' object has no attribute 'Gps' root@in_dev_docker:/apollo#

[RamboRick]

Hi yuan,

How do you make your own bag files? I mean I also want to add one obstacle based on the demo . But I don't know how to extract data from the demo files. Thank you!!