Interface user action logging

[mayacakmak]

To better understand interface usage we would like to look at descriptive stats such as how long the user is spending in each tab or which robot actions are they triggering how many times.

One way we could do this is from the front-end (running in the user's browser)--send some data to a database (with anonymous user ID that would automatically stay consistent across a user's session and separate across participants) for every interface event of interest with a timestamp. The events should be a complete list that allows us to infer all quantities we are interested in.

The other way to do this would be on the back-end side, which basically receives a command every time something of interest happens in the front-end anyways. There seems to be some code in there that might have had the same intention (e.g. all commands sent to ROS --which is not the full list we're interested in-- accumulated in commands_sent_body but does not seem to be logged somewhere). To make sure we don't loose all data if something happens during the session, we want some sort of continuous logging but I'm not sure how to do that with JS from a browser (there must be a way though).

[mayacakmak]

Okay, this is implemented in the stretch_test branch (I was able to reuse a lot of code from another project in database.js) and tested--seems to work, example data log on the database:

The only thing missing is the start and end of "tasks" -- we could guess it based on a large space between two actions or by looking at videos, but it would still be a time consuming manual process. One thought is to add a button that we ask the participant to click when they start and end a task, which would add an additional event log. Would make data processing much simpler and it is a non-consequential change in terms of the interface; we would need to explicitly ask the participant before and after every task though (i.e. don't change interface instructions in any way, don't rely on them to remember).

@csemecu @tapomayukh Let me know what you think! Also make sure to pull in the morning and test (it will make a new user for every browser so no harm in testing as you like).

[csemecu]

Those numbers in the data look very similar to the rosbag times. I wonder if there is any relation so we can make a note from the rosbag recording to check those times where the tasks begin and end. From participants 7 onwards, we have recorded smaller bags containing one task each. I will try this out in the morning

[csemecu]

I just tried the interface with my laptop as an operator twice. The first one I went through all the tabs hitting all actions with different speed selections. The second was only in the drive one checking all the speeds. I also checked that there is a topic /stretch_controller/follow_joint_trajectory/goal which we can add to a rosbag that describes the joint moving and the increment (I tried this with the drive tab and different speeds) I will try to add this topic to the rosbag of the next participant and maybe we can compare with the database entry just to be sure which method to use

[mayacakmak]

I'm working on a simple admin tool to inspect and modify the user data so we can easily add the task start/end timestamps.

[mayacakmak]

Okay here's a first version for the page that displays the logged data: https://hcrlab.github.io/stretch_web_interface/shared/data.html

Only @mayacakmak @csemecu @tapomayukh have admin permissions that allow overwriting onto user data. You need to 'login with Google' at the top to get the admin permissions. The interface allows manually changing the participant ID and adding timestamps for start/end of tasks 1-4. At the top of each user's data there is a count of how many events were logged b/w the start and end of each task, for verifying that manually entered timestamps are correct.

Since working manually with timestamps has turned out to be a pain, I still added the button onto the operator interface which toggles 'Start Task 1' > 'End Task 1' > 'Start Task 2' >... and basically just logs the timestamp for start and end of tasks. It's the first thing below the line. Pull stretch_test branch to get the updated operator interface. It is worth trying to remember to tell the participant to click it--manual data cleaning is a pain.

[mayacakmak]

This is not a high priority at the moment since the interface will change drastically but it would be good to revive the ability to record the user's interface actions at some point.

• [ ] Update logging based on latest interface, fix bugs (not properly logging at the moment)
• [ ] I keep getting a warning from Firebase that our database has insecure rules, should fix that.
• [ ] Update data visualization tool

@KaviMD

[kavidey]

@mayacakmak Ok, I can get started on those. Can you add me to the firebase, right now the only project I have access to is access-teleop

[mayacakmak]

Actually I'm in the process of adding a bunch of issues--I'll tag you in a few. I'd say this is lower priority at the moment, so you could prioritize those others.

I think this still uses access-teleop project on Firebase.

[mayacakmak]

Some problems mentioned here might need be fixed while addressing Issue #16.

[hello-ck]

I'm glad you found a solution that works for you!

For reference, with the upstream version of the web interface, the operator's browser starts recording command logs and video when the operator clicks the Start Recording button. After the operator clicks the Stop Recording button, they can click the Download Recording button, which downloads a time and date stamped .webm video, a .json command log file, and a .json sensor log file.

The command log file records all JSON commands sent from the operator's browser to the robot's browser. These commands include commands to change the mode.

I just performed a brief test with the wide-angle cameras branch of the upstream web interface. Operator recording seems to still work. However, the sensor log grows too quickly due to the new torque feedback and should be disabled. I used to be able to record over relatively long periods (30 minutes?), but I haven't tested this in a long time.

Links to relevant code follow:

This code adds each command to a log before sending it over the WebRTC data channel.

This code creates and sends mode change commands.

This code adds commands to the log.

This code begins recording with the operator's browser.

This code saves the logs.

Best wishes, Charlie