Data logging tool (continuation from #25)

[chcorbato]

Do we need a header timestamp in all our messages for data logging?

• [ ] List all publishers we have created that we need to log (from https://github.com/rosin-project/metacontrol_sim/issues/25#issuecomment-628789858)
• [ ] Add header in .msg if needed
• [ ] Make sure messages published have a correct timestamp initialized

[chcorbato]

@marioney does this issue makes sense to you? At least I know that I have not been taking care of header/timestamp in my action publishers in mros1_reasoner

[marioney]

I'm not sure about this.

* Add header in .msg if needed

You mean to publish timestamps on the "sensor" messages (i.e. power_load and dist_to_obstacle) We can do this, but I'm not sure if the observer will need it / handle it.

For the interactions with the reasoner, I agree we need a correct timestamp, but I guess that's taken care of by the graph_observer

[chcorbato]

Maybe @gavanderhoorn has advise on logging ROS info at runtime

• what logging freq?
• better to log the time info in the messages?

[gavanderhoorn]

* what logging freq?

This question is a somewhat strange in an event based system?

* better to log the time info in the messages?

If you're interested in capturing the time of generation or (almost) publication, then add a header.

But std_msgs/Header is so small, I would almost suggest to always add it to messages which encode sensor data (or the results of processing sensor data, ie: derived data).

[chcorbato]

@gavanderhoorn do you have a recommendation for a tool to log and process the data?

[gavanderhoorn]

What is the purpose of the log?

Complete reproduction of the message flow, or exporting only 'interesting' streams to external tools for (post)processing (ie: plotting, etc)?

[marioney]

Complete reproduction of the message flow, or exporting only 'interesting' streams to external tools for (post)processing (ie: plotting, etc)?

In this case is the second option, we just need to compare the "performance" of two different approaches

[chcorbato]

We are running a set of experiments for the navigation (just move_base) in a version of the MOOC factory simulation, with and without metacontrol, to measure performance through a set of variables (see https://docs.google.com/document/d/1FRyrRr9ETblZj5e3-rSsoi8XIl6d_fzOLu6jlB4bxLY/edit#heading=h.65kwbgg1ysfd) This will require some post-processing. Visualization can be a separate tool, since it will be graphs for the paper, although can be useful to understand what is going on with the experiments. We could be using IPA's ATF, but we have a week to submit the paper with the results

[gavanderhoorn]

Ok.

Then I would just start with rosbag. If it turns out you don't get the performance you need, there are other tools to try.

For getting the messages/making plots, I'd then recommend PlotJuggler and/or some simple Python scripts with the rosbag API: rosbag/Cookbook.

You would not be using rosbag play here.

[gavanderhoorn]

Oh and be sure to check which topics you'd need / want to record. Unless of course you don't have any high-bandwidth topics. Then you just record everything.

Note that messages with Headers would always be good in these cases, as otherwise you can only use the time rosbag received the message for your plots, which will result in very jittery graphs.

[gavanderhoorn]

@marioney wrote:

I'm not sure about this.

* Add header in .msg if needed

You mean to publish timestamps on the "sensor" messages (i.e. power_load and dist_to_obstacle) We can do this, but I'm not sure if the observer will need it / handle it.

Does the observer ever have to (cor)relate sensor data to other data? Or to previous samples? Are derivatives ever calculated?

If so: add Headers.

Never use ros::Time::now() or wall clocks.

---

And for your plots the Headers are also good to have, as I mentioned in my earlier comment.

[marioney]

Then I would just start with rosbag. If it turns out you don't get the performance you need, there are other tools to try.

The problem with rosbag is that we want to do thousands of experiments, and then get statistics / analysis from them, so a lot of post-processing will be required.

Maybe we should have started with this, but all these questions are because I'm working on a "logger" that directly subscribes to the interesting topics and logs the data in a .csv file

[gavanderhoorn]

The problem with rosbag is that we want to do thousands of experiments, and then get statistics / analysis from them, so a lot of post-processing will be required.

I'm not sure how "thousands of experiments" changes things here. You imply there are problems that come up when doing those nrs of experiments.

Can you describe what the difference in workload is between writing a custom implementation and using the infrastructure I described?

Rosbag can be started as part of a .launch file.

The Rosbag API makes it almost trivial to extract the data you want (and the convert it to whatever you want, if needed).

Maybe we should have started with this, but all these questions are because I'm working on a "logger" that directly subscribes to the interesting topics and logs the data in a .csv file

yes, that does sound like duplicated effort as this is essentially what rosbag does.

Unfortunate.

[marioney]

I understand your point, and I agree that we can use the rosbag infrastructure to get the data.

I'm not sure how "thousands of experiments" changes things here. You imply there are problems that come up when doing those nrs of experiments.

I'm not saying that the problem is the number of experiments, or how to record the bags, it was more about how to process the data (i.e. compute distance to goal, count number of goals, ect).

Can you describe what the difference in workload is between writing a custom implementation and using the infrastructure I described?

The two options are: store the rosbags and then create a script to extract the data as .csv vs create a script to subscribe to the topics and extract the data as .csv, hence the question about a logging tool that initiated this thread.

[chcorbato]

the question about a logging tool that initiated this thread.

Things got messed up, probably my fault. I changed the issue name to point that this is a con. of #25

[gavanderhoorn]

Personally I'd probably prefer to spend my time on writing a script which processes the data rosbag has collected for me, as rosbag already takes care of storing, splitting, etc, etc of the data. Things I don't want to have to deal with.

But it probably depends on where you have the most experience and what you're comfortable with.

The two options are: store the rosbags and then create a script to extract the data as .csv vs create a script to subscribe to the topics and extract the data as .csv,

Well, what about using rosbag to record the data and then write a script which takes that data and immediately produces plots? Skip the .csv.

Unless you need .csv for something else of course.

In any case: use what you're comfortable with.

[marioney]

In the end I'm using a program I developed to stop the simulation and get some quick data in .csv

Well, what about using rosbag to record the data and then write a script which takes that data and immediately produces plots? Skip the .csv.

That's a good option, I'm recording the bags in any case, as you said recording the bags is just one more line in the launch file.

[chcorbato]

See https://raw.githubusercontent.com/rosin-project/metacontrol_experiments/master/run_batch_sim.sh