search

PhotonVision / photonvision

PhotonVision is the free, fast, and easy-to-use computer vision solution for the FIRST Robotics Competition.
https://photonvision.org
GNU General Public License v3.0
279 stars 199 forks source link

Enhanced Tooling for SIL support #864

Open gerth2 opened 1 year ago

gerth2 commented 1 year ago

Documenting discussion starting here in discord.

A useful piece of functionality would be allowing photonvision to better function with a simulated robot. There are two conceivable ways this could happen:

"SIL-SIL" - both robot code and photonvision are running on a development desktop, with no real robot hardware at all.

"SIL-HIL" - Robot code is running on a development desktop, but photonvision is running on other hardware (presumably the same coprocessor that would be used on a robot)

In both cases, a common functionality need would be to have photonvision accept camera input from an alternate stream (possibly mjpg stream, since cscore supports it?) that was networked, rather than hardcoded to a certain physical camera. In turn, this would allow the simulation environment (advantagescope, Unity, gazebo, xRC, some custom java code, really anything) to submit simulated frames into photonvision for processing.

In the SIL-SIL case, presumably, there would be some workflow by which a user would have some config shared between their development desktop and their real-world coprocessor. There would need to be startup flags passed to photonvision so it knows whether to use the configuration verbaitum, or to (optionally?) swap out some cameras for the alternate stream.

In the SIL-HIL case, simply configuring alternate camera types (but retaining same name) might be sufficient, as it's most reasonable that users would have a second set of hardware other than their robot.

Open questions: Are these workflows realistic? What other factors have we not considered? what other tooling is useful for facilitating these workflows? Is there a "one true way" of doing simulation which we can tell users to reduce the number of development choices they have to make?

amquake commented 1 year ago

Additionally, if the camera stream is provided by an external simulation environment, could camera settings (resolution, gain, brightness, etc) be communicated?