Controller Movement Visualization: Implement First Version With D3

[bastianilso]

For the Whack-A-Mole VR dashboard, we want to visualize how players have moved their controller while playing. I made a sketch of how we envision this below:

The idea is to try to implement this in D3:

• The Whack-A-Mole wall will be a background image to the D3 visualization.
• The visualization itself is a rectangular surface with x axis from -2 to 2 and y axis from -2 to 2.
• We should start with visualizing movement activity as simple "dots" within this rectangular surface.
• On top of the rectangular surface we draw the player and a VR controller.

You can find the image assets in here (if possible, you can use the vector graphics SVG files directly in D3?) movement-vis.zip

[bastianilso]

Here is some example data (it's 20 seconds of movement by a player in Whack-A-Mole VR). You need to use the columns RightControllerPosWorldX and RightControllerPosWorldZ.

data.csv

[aldsanms]

For the Whack-A-Mole VR dashboard, we want to visualize how players have moved their controller while playing. I made a sketch of how we envision this below:

The idea is to try to implement this in D3:

• The Whack-A-Mole wall will be a background image to the D3 visualization.
• The visualization itself is a rectangular surface with x axis from -2 to 2 and y axis from -2 to 2.
• We should start with visualizing movement activity as simple "dots" within this rectangular surface.
• On top of the rectangular surface we draw the player and a VR controller.

You can find the image assets in here (if possible, you can use the vector graphics SVG files directly in D3?) movement-vis.zip

---

*Here is a screenshot of the CSV file data visualization:

data.csv

The data is read by the R program and processed in the JS. a background image shows us the axes of the graph. this image follows the size of the window in order to maintain the correct values of projection points.

I haven't managed to put a local background image yet, so for the moment I got an image from the web.

The visualization itself is a rectangular surface with x axis from -2 to 2 and y axis from -2 to 2.

*This is the R program:

library(r2d3)

AllData <- read.csv(file = "C:/StageA4/Rstudio/test/02/data.csv", stringsAsFactors = FALSE, strip.white = TRUE, sep = ';')

r2d3( data=AllData, script = "C:/StageA4/Rstudio/test/02/test01.js" )

*And this is the JS program:

var barWidth = Math.ceil(width/2); var barHeight = Math.ceil(height/2);

var backgroundPicURL = "..."

svg.selectAll('rect') .data(data) .enter().append('rect') .attr('width', 2) .attr('height', 2) .attr('y', function(d) { return barHeight + (d.RightControllerPosWorldZbarHeight)/2}) .attr('x', function(d) { return barWidth + (d.RightControllerPosWorldXbarWidth)/2}) .attr('fill', "blue");

svg.attr('style', "background: rgb(55, 255, 255);background-image: url("+backgroundPicURL+");background-repeat: no-repeat;background-position: center;background-size: 100% 100%;")

[bastianilso]

Few comments 👍

• It should be possible to read your D3 script and CSV file using a relative file path instead of an absolute path. (if the R project and your files are located in the same folder). I believe this should also be possible for the background image.
• The JS program you are showing, is that using D3? (I dont see a mention of D3 in there, hence why I'm asking). You may need to use a D3 scatterplot example as your basis (I see variables like barHeight and barWidth in there, which I assume are leftover from the previous barchart exercise)

[aldsanms]

Few comments 👍

• It should be possible to read your D3 script and CSV file using a relative file path instead of an absolute path. (if the R project and your files are located in the same folder). I believe this should also be possible for the background image.
• The JS program you are showing, is that using D3? (I dont see a mention of D3 in there, hence why I'm asking). You may need to use a D3 scatterplot example as your basis (I see variables like barHeight and barWidth in there, which I assume are leftover from the previous barchart exercise)

---

The script in the JS file is a D3 script.

For barWidth and barHeight I need it for point positioning.

[aldsanms]

Here is the error i get when i try to load a local image.

I tried with several paths and with "/" or "\", I always have the same message.

[bastianilso]

@aldsanms what happens if you use a reltative path? say just axe.png ?

[aldsanms]

@aldsanms what happens if you use a reltative path? say just axe.png ?

It sends the same message.

[bastianilso]

@aldsanms did you try without using "file:///" ? ( I just googled your error and found that link below).

https://stackoverflow.com/a/37690646

[aldsanms]

@aldsanms did you try without using "file:///" ? ( I just googled your error and found that link below).

https://stackoverflow.com/a/37690646

It does the same thing

[bastianilso]

@aldsanms that path in your screenshot still contains file:///.

[aldsanms]

OK so it works with Shiny.

[aldsanms]

Here is the result of my first test with Shiny and R2D3:

https://user-images.githubusercontent.com/73821252/192461670-10d8f6fb-3811-43c5-9227-03b8f387877f.mp4

You can see the code here : -> link <-

With this code, we can know when an element is hovered or clicked, and then perform an action afterwards.

[bastianilso]

Hi @aldsanms nice! We will explore more interactivity later too.

I put a link to a zip file in the original message with image assets for you to create the Controller Movement Visualizization. Could you download this and create the Controller Movement Visualization in R Shiny?

https://github.com/med-material/d3-rshiny-vis/files/9624584/movement-vis.zip

[aldsanms]

I put a link to a zip file in the original message with image assets for you to create the Controller Movement Visualizization. Could you download this and create the Controller Movement Visualization in R Shiny?

I'm looking for this now.

[bastianilso]

@aldsanms I have updated the image assets to accomodate for movement in both negative and positive X and Y, see mockup below.

you can download the updated assets here: movementvis.zip

[bastianilso]

@aldsanms if you want, you could animate the VR controller moving around, as if the person was playing. I will send the circle and the controller as two separate assets: separate.zip

[bastianilso]

@aldsanms an additional feature you can explore: tracking mouse X and Y movements within the visualization and show the numbers in terms of units on the X and Y axis underneath:

[bastianilso]

I think drawing some basic grid lines to mark "1 meter" and "-1" meter in X and Y directions can also help reading the graph.

[bastianilso]

The head on the visual mockups are not for scale, by the way. The head should ideally be ~20 cm big to correspond approximately to the size of the human head.

[bastianilso]

Remaining todos for this visualization:

1. Replace background image with D3 drawn lines.
2. Zoom the graph a little so that the outer edges are 1.25m, and -1.25m, so we can see data better.
3. Replace "red dot" with controller icon, and make controller move around.
4. Introduce text underneath the graph, where we can write X and Y coordinate of the mouse cursor.
5. Introduce scrubbing/playback.

[aldsanms]

Remaining todos for this visualization:

1. Replace background image with D3 drawn lines.
2. Zoom the graph a little so that the outer edges are 1.25m, and -1.25m, so we can see data better.
3. Replace "red dot" with controller icon, and make controller move around.
4. Introduce text underneath the graph, where we can write X and Y coordinate of the mouse cursor.
5. Introduce scrubbing/playback.

---

Today I managed to realize almost all the issues:

1. Replace background image with D3 drawn lines.
2. Zoom the graph a little so that the outer edges are 1.25m, and -1.25m, so we can see data better.
3. Introduce text underneath the graph, where we can write X and Y coordinate of the mouse cursor.
4. Introduce scrubbing/playback.

(no additional libraries are needed)

Also, I added other features:

A. We can zoom in real time to the center of the graph. B. The axes and the point representing the head are automatically resized in real time. C. All parameters, values and sizes are easily editable with variables.

I still have to do:

3. Replace "red dot" with controller icon, and make controller move around. -> Fix bugs (red dot jumps when zooming)(crashes when zooming in or out too much)

Here is a presentation video:

https://user-images.githubusercontent.com/73821252/194073584-0c5ddfbe-22ca-451f-97e1-23b955955ea4.mp4

[aldsanms]

Today i changed the background image of the controller and added a move function in the graph.

https://user-images.githubusercontent.com/73821252/194332283-ccd0da62-3f95-4319-82bb-0ae16082eb88.mp4

[hendrikknoche]

hmm, can we use the orientation for the controller? it looks a bit strange when it doesn't rotate naturally.

[aldsanms]

hmm, can we use the orientation for the controller? it looks a bit strange when it doesn't rotate naturally.

Yes it should be possible, i will look tomorrow.

[aldsanms]

I managed to change the rotation of the controller, I still have some calibrations to do.

https://user-images.githubusercontent.com/73821252/194557920-251c5d48-d251-4a8a-a6b1-3e76a85f7208.mp4

[aldsanms]

I managed to generate moles with different ID than number of columns and rows stored in variables. It is now easy to change some moles arguments like colors or size.

https://user-images.githubusercontent.com/73821252/194836432-909ea5d9-43f5-4c12-8d1f-25411180b5a6.mp4

[bastianilso]

• Use the RightControllerLaserPosTravelX for moving a laser around on the wall. Should be shown simulatenously to the person playing.
• Use #b3b3b3 for the grid lines and draw them underneath the person and the data.
• Prepare for a code review - where we go over the architecture of your visualization and let me know when you think it's ready.

[aldsanms]

Today I managed to:

• Show laser movement.
• Show controller movement.
• Show the moles.
• Change the states of moles on the wall.

https://user-images.githubusercontent.com/73821252/198294426-cee6e338-1e3c-46b4-83e1-9aa1f51c3539.mp4

I still have to do:

• Controller rotation.
• Time bar scroll.
• Add axis.

[bastianilso]

@aldsanms really cool, well done! Looking forward to try this with the time scroll bar.

[aldsanms]

Today I managed to:

• Controller rotation.
• Time scrollbar.

https://user-images.githubusercontent.com/73821252/198595394-9615aea9-6182-44ca-a404-8b8436f2cea8.mp4

I still have to do:

• Add axis.
• Add a time caption below the scrollbar.
• Sometimes moles don't update properly and stay in false state after releasing the scrollbar.

[aldsanms]

Today I managed to:

• Solve the problem of moles that no longer disappear.
• Add player field of view.
• Fixed bugs when scrolling the scrollbar.

In this video time is accelerated. In this example we can see that when the mole is not in the player's field of vision, he does not click.

https://user-images.githubusercontent.com/73821252/199253448-0620ce25-7146-4778-b2dd-3ee383d53b9e.mp4

I still have to do:

• Add axis.
• Add a time caption below the scrollbar.

[bastianilso]

Hi @aldsanms, the head orientation plot and the controller visualization plot are supposed to exist as two different visualizations, it would be better if you post updates regarding the head orientation plot in issue #5 👍

[bastianilso]

@aldsanms let me know what the status of the controller movement visualization. When you feel the controller movement visualization is ready to be reviewed, create a pull request for it.

[aldsanms]

Today I finished the functions needed for display and movements. I can't display the time axis below the scrollbar as I want.

Here is an example: