GRIP SmartDashboard Extension

[rlee287]

Right now, teams can view a video feed of any image in the SmartDashboard. However, it would be nice to be able to preview the ContoursReport, LinesReport, etc. graphically on the Dashboard as well.

[bathtubed]

I feel this would be suited as a Smart Dashboard extension. Have a widget that takes a video stream and a Contours or Lines Report and renders them over the stream image.

[ThomasJClark]

@EdWard680 sounds good. I think the extension could also be used to get video from sources other than the roboRIO (the USB cam viewer is hardcoded to use the RIO, so GRIP can't publish to it if it's running on a coprocessor). I might look into this if I have some time

[gartaud]

@ThomasJClark We have installed the latest version of GRIP and the Smart Dashboard extension. It is a very neat concept but we frequently get "Invalid stream (wrong magic numbers)" messages (and therefore the display flickers a lot). Have you observed such issue?

[ThomasJClark]

What device you running GRIP on? If you're running on the RIO, try using the built-in SmartDashboard USB Camera viewer and seeing if the same problem happens.

What are you Publish Video settings?

Can you get a wireshark capture?

[gartaud]

We are running both GRIP and the Smart Dashboard on a Windows 10 PC connected to an Axis camera (over IP). Streaming directly from the camera to the Smart Dashboard works perfectly fine. Streaming from the camera to GRIP also looks fine. But streaming from GRIP to the GRIP extension is problematic. I believe that the Publish Video Settings are the default (quality 80%) but I would have to double-check. I suppose we could do a wireshark capture tomorrow. Do you think that changing the quality settings could make a difference? Thanks!

[ThomasJClark]

Yeah, see what happens if you turn down the quality. Although, It should really be fine if it's on the same machine.

Can you also try using the built-in USB Camera viewer?

[gartaud]

OK, we will try turning down the quality as well as the built-in viewer tomorrow and update you. Thanks!

[DrewTheRat]

I had opened a separate issue on this, but it looks like the conversation is going on here.... @ThomasJClark - we are using an Axis web cam, not USB. What are you requesting we look at with the USB viewer? For the WireShark capture, what traffic are you interested in? The plugin is talking to local GRIP, so that would be traffic on the loopback. Or do you want traffic between the laptop and the webcam?

[gartaud]

I suspect that "USB" was meant to say that we should look at traffic coming from the laptop itself using the standard Vision module (possibly View, Add, Laptop Camera instead of Camera?) although right-clicking on the camera when in the dashboard is in editable mode (View, Editable) should allow to change the IP address of the camera from the default 10.TE.AM.11 to localhost (assuming that GRIP behaves exactly as a camera)? But all this is speculation... I am confused too.

[ThomasJClark]

@ThomasJClark - we are using an Axis web cam, not USB. What are you requesting we look at with the USB viewer?

"USB Webcam Viewer" really just means the protocol that the roboRIO uses to stream video to the DS. GRIP uses the same protocol, so if you're running GRIP on the RIO, you can use that viewer. This will let us know if there's a problem in GRIP or a problem in the GRIP SmartDashboard extension.

[gartaud]

But what if we are running GRIP on the same PC as the Smart Dashboard? Can we specify an IP address for the USB Webcam viewer?

[bathtubed]

You can't point USB Web Camera Viewer to anything else other than the roborio address, but you can point the GRIP Viewer to other addresses, like localhost

[DrewTheRat]

We are using a webcam (Axis 1011) and not a USB camera. Originally we were viewing the camera's video stream using the VideoStreamViewerExtension widget. That had no issues with the video stream from the Axis, so we don't think that's the issue....

[ThomasJClark]

We are using a webcam (Axis 1011) and not a USB camera. Originally we were viewing the camera's video stream using the VideoStreamViewerExtension widget. That had no issues with the video stream from the Axis, so we don't think that's the issue....

• VideoStreamViewerExtension uses a standard M-JPEG stream and lets you specify an address, so it works directly with an IP camera.
• WebcamViewerExtension uses the FRC dashboard protocol and is hardcoded to the roborio address, so it works when you're using CameraServer or GRIP on the roboRIO.
• GRIPExtension uses the FRC dashboard protocol and lets you customize the address. So, it can work with GRIP or CameraServer regardless of where the images are coming from.

@gartaud Try:

• using the GRIP extension to view a regular CameraServer feed from the roboRIO
• using the USB webcam viewer to view a stream from GRIP (if you can run GRIP on the RIO).

If the first works and the second doesn't, it's a bug in GRIP's "Publish Video" step, and if the second works and the first doesn't it's a bug in the GRIP SmartDashboard extension.

[gartaud]

Hi and thank you for taking the time to list all the options. To the best of my knowledge we do not run CameraServer on the robot (as we used to directly talk to the camera over IP). Where can we find more info about CameraServer? (and we do not currently run GRIP on the robot itself either by I think we have enough information to try) Thanks!

[gartaud]

@ThomasJClark I can reproduce the issue by opening a static image in GRIP (a jpeg picture taken with a point and shoot digital camera), and then doing a resize image, followed by a RGB threshold on the resized image, a find contours on the output of the threshold, a publish video on the resized image, and a publish contours report on the contours. If the resize is aggressive (down to 160X120) I can go up to 100% on quality and do not see any error in the grip extension. If the resize leaves 320X240 I see the error pretty constantly above 95% quality At 640X480 it pops up as early as 40% quality At 1280X960 I cannot go past 5% quality So does this error simply mean that the frames cannot be sent fast enough or is it indeed a bug on the publish video (which you should be able to reproduce easily using a fixed jpeg picture, isn't it)?

[ThomasJClark]

It sounds like the error is that frames can't be sent fast enough or aren't being processed fast enough. I'll look into it more and see if I can reproduce it.

[gartaud]

Do you need a grip file or is the description of the imaging chain good enough?

[ThomasJClark]

The description is enough

[gartaud]

OK. Thank you! Note that the PC were I observed that is fairly old/slow by today's standards (Core 2 Duo E6550, 4 GB RAM, Windows 10 x64)

[gartaud]

Hi. Any luck reproducing the issue?

[ThomasJClark]

Yeah, I think I fixed it. It was a minor bug in the extension. Let me know if the new release works:

https://github.com/WPIRoboticsProjects/GRIP-SmartDashboard/releases/tag/0.1.1

[gartaud]

Thanks we will try within the next few hours and get back to you.

[gartaud]

@ThomasJClark It looks like the old message is gone but we now get "No image available" or "Connection reset" from time to time instead (although probably not as frequently as we were getting "Invalid Stream"). Is it expected? I cannot fully reproduce with a static image, but the following might be a clue: If you close a smart dashboard layout file and reopen it, it will say "No image available" until you change a setting in the GRIP imaging chain. Also if you put the GRIP extension panel in the top left corner of the smart display and click on File in the menu so that the menu obscures part of the panel, and then close the menu, then the part of the image that was under the menu frequently becomes pinkish and says "No image available"

[DrewTheRat]

@ThomasJClark, adding to what @gartaud reported: When we were testing tonight, the plugin was flashing rapidly between the video stream with contours and the pinkish background with the "Connection reset" message. However, if we moved the camera to a position where no contours were detected, the flashing stopped.

[gartaud]

No fix?

[ThomasJClark]

@ThomasJClark It looks like the old message is gone but we now get "No image available" or "Connection reset" from time to time instead (although probably not as frequently as we were getting "Invalid Stream"). Is it expected?

Glad to hear it's working a little better.

I'm having trouble reproducing that regardless of how high quality the image is. How frequently does it happen? Also, are there any errors in GRIP.log in your home directory?

I cannot fully reproduce with a static image, but the following might be a clue:

Yeah, I'm actually not surprised the extension doesn't fully work with completely static images. Publish Video basically only send images when something changes or it gets a connection for the first time. This works fine for streams of video, but probably not always for static images, since they'll only change if you change to a different image or mess with the pipeline at all.

Also if you put the GRIP extension panel in the top left corner of the smart display and click on File in the menu so that the menu obscures part of the panel, and then close the menu, then the part of the image that was under the menu frequently becomes pinkish and says "No image available"

Screenshot? I can't reproduce this either.

Also, is your networktables server OutlineViewer or the roboRIO? Most of my testing has been using OutlineViewer, so that may be related.

[gartaud]

Good morning

I'll try to clarify: On the Windows 10 x64 laptop where we run GRIP (to compute the contours and re-publish the video) connected to an Axis IP camera on the robot (not involving roboRIO whatsoever) and the Smart Dashboard with the latest version of the GRIP extension we get "No Image available" or "Connection reset" fairly frequently, especially (and possibly only) when contours are found (and when the potential targets are highlighted). We will have to check the GRIP.log

On the Windows 10 x64 desktop where we run GRIP using a static image (attempting to find a deterministic way to reproduce the issue seen on the laptop) I am not successful at reproducing the exact same problem but you can easily suppress part of the display by opening the menus of the Smart Dashboard

A screenshot can be found at: https://onedrive.live.com/redir.aspx?cid=628f1d3c64133a16&resid=628F1D3C64133A16!153463&parId=628F1D3C64133A16!118&authkey=!AlgqhS4Xw4eYisA&Bpub=SDX.SkyDrive&Bsrc=Share&ref=button&type=1

Thanks, Gabriel

[gartaud]

Hi @ThomasJClark . Is the screenshot helpful? Thanks.

[ThomasJClark]

I understand now, but I still can't reproduce the error on my machine, so I'm not sure what's wrong.

Could you upload the GRIP.log? GRIP itself might be running into problems.

[gartaud]

I don't see anything surprising in the log. Well I guess we will get back to you if we ever get more info to share. Thanks!

[jabackma]

Gartaud, we are running an identical situation to yours last year and see dropouts for about 1 second, every 10 or 20 seconds. Did you ever resolve your issue? We are using an i5 but it 'usually' seems to work ok.

John

[gartaud]

@jabackma We are not using GRIP this year (we are using OpenCV directly on a Pi computer on the robot and we stream videos and contours from them)... but it was eventually mostly working OK for us last year. Are you using the Grip Extension? If so are you using the latest? Do you get an error message or just flickering? If you are using the smart dashboard are you sure nothing is overlaying the display of the images (e.g. no control at the same location/or too close to the images)... and that the display is fully within the smart dashboard window (i.e. not too close from the edge)?

[jabackma]

Thank you for responding Gartaud, yes we are using the plugin for the driver station the, Grip Output Viewer. All we get is a flickering for about a second to a pink screen, then back to operation. Did you find that the driver station other items were affecting the drawing? The images shown in Grip do not flicker, only the one that's shown on the driver station.

When we get the flicker, the Publish Video time goes to 0 in Grip.

I guess our backup will be to use Grip itself as the image the guys use to look at during competition.

[gartaud]

We found once that we had two instances of the Grip extension on top of each other in the smart dashboard and that created problems (only the top one was visible so it was not obvious to troubleshoot). Do you make sure that you subsample the images you are analyzing so that it does not take too much processing time and bandwidth (e.g. to 320*240)?

[FRC2495]

Is the GRIP SmartDashboard Extension still supported?