benloh commented 2 years ago

Merges dev-bl/webcam -> dev-next

WebCam

This adds a background div behind the simulation render with a video stream from a camera attached to the server.

Currently only tested on Chrome. You'll need to allow permissions to access the camera. You should be able to select the camera source using Chrome's "Privacy and security" > "Site settings" > "Camera".

Activating

The "WebCam" checkbox on Main (navbar) will turn the camera on and off.

Calibration

When the camera is on, you can use the sliders to change:

Scale
Translation (x/y offset)
Rotation
Mirroring (horizontal/vertical) The settings are saved immediately to the project file and will be loaded the next time you load the project. (This includes the webcam on/off status).

Because each project can specify its own simulation world size (aka "boundaries"), the webcam settings are specific to each project. This means that you will need to load and adjust the camera for each game. Advanced users can just copy and paste the relevant metadata values from one project to another if the projects are the same size. If the projects are not the same size, the mappings may be off.

How It Works

Basically this works by adding a

PanelSimulation places the video canvas behind the simulation. This is necessary so that the agent instances remain selectable by the mouse.

Project Design Considerations

The layering of the video UNDERNEATH the simulation layer means that the simulation layer needs to be transparent. This has some serious implications for project design:
If a project is using a solid color background, we render that background with a 50% alpha so that the video is visible underneath.
This will be a problem for projects (like decomposition) that use a solid background image. The video will be hidden on these projects. The workaround is set alpha to 0.5 on background agents.

To Test

Connect a webcam to your server computer (or you can use the built-in camera)
Start Chrome
In Chrome settings "Privacy and security" > "Site settings" > "Camera"...
...select the camera you want to use.
...click "Sites can ask to use your camera"
You'll probably also want to set up and run either pozyx or ptrack for calibration purposes.
npm run gem
Go to http://localhost/app/login and select a simple project with no background sprites
Go to Main
Click the "WebCam" checkbox.
If you see The user denied permission to access the web camera. in the web console, then you did not enable the camera correctly.
You'll probably be asked for permission to access the camera: "http://localhost wants to: > Use your camera". Click "Allow"
After about a second the stream should start.
Move a Pozyx tag or PTrack object/person onto the play area. Generally it's easiest to set up 3 points for comparison.
Use the sliders at the bottom of the screen to adjust the video to map to the person/objects.
Your settings will be saved with the project so that next time you load the project, the project will retain the same calibration. The project will also remember if the WebCam checkbox was on or off.

If you want to test background sprites, currently the solution is to edit the initScript for the background sprite, adding the line: prop alpha setTo 0.5. For some reason setting it in the blueprint does not currently work. Looking into that.

Known Issues / To Do

[x] Figure out how to set alpha on background sprites so video will show through
- [x] initScript works, but not setting it in blueprint
[x] @jdanish Decide if having video on back layer is OK (e.g. you have to adjust background sprites), or if you want video on the top layer and give up the ability to select and interact with the agents.

benloh commented 2 years ago

added 1 commit

16a65d95 - webcam: Update renderer as WebCam checkbox is toggled on and off. Necessary...

Compare with previous version

benloh commented 2 years ago

requested review from @jdanish

benloh commented 2 years ago

In GitLab by @jdanish on Sep 14, 2022, 14:36

I think we need the video on the back layer so that we can move things around, etc. But I'll verify with Noel once I can test.

benloh commented 2 years ago

added 3 commits

6b2c4e53 - webcam: Convert 'METADATA_LOAD' message into more specific 'WEBCAM_UPDATE' so...
0d46130b - webcam: Doc
fa7540d9 - webcam: Export UpdateMedaData on project-server.

Compare with previous version

benloh commented 2 years ago

added 3 commits

b14c4250 - webcam: Convert METADAT_LOAD to WEBCAM_UPDATE (continued)
af4fd927 - webcam: Revert "Tracker" and "Export" links to buttons so they're more easily seen.
de7592b3 - webcam: Set default webcam metadata values. Fixes react warning about controlled inputs.

Compare with previous version

benloh commented 2 years ago

added 2 commits

13df6800 - webcam: Properly store and retrieve mirror values. Fixes bug where Mirror...
7cd68f10 - webcam: Properly track `streamStarting` state. Fixes bug where multiple...

Compare with previous version

benloh commented 2 years ago

added 1 commit

fadbeb9b - webcam: Prevent multiple render loops from being initiated when toggling...

Compare with previous version

benloh commented 2 years ago

added 1 commit

29f35134 - webcam: Properly handle missing parameters when calling SetGlobalConfig. Was...

Compare with previous version

benloh commented 2 years ago

added 1 commit

5c507510 - webcam: Use absolute positioning for renderer to fix webcam offset.

Compare with previous version

benloh commented 2 years ago

added 1 commit

9de75b93 - help: EditSymbols now properly look up feature methods instead of falling back...

Compare with previous version

benloh commented 2 years ago

marked the checklist item Figure out how to set alpha on background sprites so video will show through as completed

benloh commented 2 years ago

marked the checklist item initScript works, but not setting it in blueprint as completed

benloh commented 2 years ago

marked the checklist item @jdanish Decide if having video on back layer is OK (e.g. you have to adjust background sprites), or if you want video on the top layer and give up the ability to select and interact with the agents. as completed

benloh commented 2 years ago

@jdanish Setting the alpha property in the blueprint for any background sprites does actually work. You just need to keep in mind that hovering over any sprite makes it opaque. This was necessary to make the selection outline be visible.

So if this works for Noel, this PR should be ready.

benloh commented 2 years ago

In GitLab by @jdanish on Sep 15, 2022, 15:20

Awesome. He should be testing it soon.

benloh commented 2 years ago

added 1 commit

35e99036 - webcam: Doc

Compare with previous version

benloh commented 2 years ago

added 1 commit

f3b16b60 - bug: Fix "no feature named Physics" bug when deleting agent instances. Fixes #438.

Compare with previous version

benloh commented 2 years ago

added 1 commit

445ebafd - webcam: Restore add instance functionality. Global agents needed to re-use existing ids.

Compare with previous version

benloh commented 2 years ago

added 2 commits

ec608916 - 1 commit from branch dev-next
c684b411 - Merge commit 'dev-next' into dev-bl/webcam

Compare with previous version

benloh commented 2 years ago

Merging.

benloh commented 2 years ago

mentioned in commit b8fd6b8369e9f3df4e6eca5792b68cff1df8204d

theRAPTLab / gsgo

Feature: WebCam - [merged] #718

Activating

Calibration

How It Works

Project Design Considerations

To Test

Known Issues / To Do