SDK v4 breaks Vuforia in landscape mode

[Windwoes]

The title pretty much says it all. It was not mentioned in the changelog, but in SDK v4.0, the Vuforia library has been updated. This seems to have helped the problem of the camera being dark in low light - which is good - but it also breaks Vuforia in landscape mode. Have a look at the following screenshots. Please note that the problem in SDK v4 is not just the image being skewed - as far as I can tell, it also screws up the positional lock as well (the lock marker will drift off the picture when moving the phone around)

SDK v3.5

SDK v4.0

[shelbyRobots]

I have noticed similar results. I copied the ConceptVuforiaNavRoverRuckus.java file and ran it. If you start holding the RC in portrait and then turn it to landscape and back, you will see the preview area change size to a long skinny aspect ratio. On a loosely associated topic, would it be possible to move the network and robot status text to the black bar - placed just below the RC phone ID? Perhaps git rid of the "Active Configuration" label in the red included header bar and put the opmode there? Also possibly relocate the gamepad 1/2 text (even laid over the image like the error text). This allows for a larger area for the camera preview to display, and helps with aspect ratios. I have tested some of this, and it works with both Vuforia and OpenCV.

[Windwoes]

@shelbyRobots we need to wait till the forums open to be sure, but I'm pretty sure it's allowed to modify the GUI layout file however you like. FTAs might not want to assist you if you have problems, but honestly now that there's no longer a break between auto and tele, there's not really anything an FTA can do that you can't, at least for experienced teams.

[KevinEmery]

@shelbyRobots I would be wary of RS06 when talking about modifying the Driver Station app, and whether or not the modifications violate it being the "official" app. I'd definitely ask on the forums.

\<RS06> Driver Station App - Teams must install the official “FTC Driver Station” app onto their Driver Station Android Device and use this app to control their Robot during a match. The Driver Station software version number must match the version number of the Robot Controller app.

[AlecHub]

The SDK is concerning the Robot Controller App. So \<RS06> does not apply here.

\<RS9> Software Modification - Teams are not allowed to modify the FIRST Tech Challenge Driver Station application or Robot Controller SDK in any fashion.

[gearsincorg]

Some points: 1) The Vuforia preview is displayed on the Robot Controller, so any modifications of the GUI would be done there, which is permitted.

2) As a rule, Vuforia will produce unexpected results if you rotate the camera AFTER it has initialized. It's always been this way. On a real robot this should never occur if the camera is hard mounted. When testing, just start the OpMode with the phone in the orientation you intend to run it on the robot.

[Windwoes]

@gearsincorg both of those points are unrelated to the issue that prompted me to open this issue, though

[shelbyRobots]

Thanks for the responses. @FROGbots-4634 I wonder if #2 is really unrelated. I have seen the skewing (or perhaps scaling in Y) that your example shows. On a subsequent run, it will be gone. So maybe it has to do with a change in the view surface during layout that is similar in concept to what happens on a layout update caused by a rotation change. As far as #1 - I understand about being allowed to change the layout on the RC. I just try to avoid changing anything outside of TeamCode other than perhaps gradle files. I was making a suggestion that I thought might benefit other teams if the change was made in the upstream layout file. It doesn't look like the real estate in the top bar is being used for anything, so this is a simple change. As for the gamepad text - I am not crazy about it living on top of the image, but most of the time we are looking at the preview in auton, so the gamepads aren't being used.

[gearsincorg]

I'm checking this out.

One thing I noticed is that you are letting your phone auto rotate. I typically have my phone locked into portrait mode, so when you run the Vuforia OpMode you get a much better viewing window in the landscape position.. I'm not saying you should Have to do this, but in the interim at may make your testing easier (and more predictable).

[gearsincorg]

What I can tell so far is that when the screen is in landscape mode, the vertical pixel scaling of the preview image is incorrect (too large). This explains why the target center seems to drift as you move the camera.

The RGB lines are being displayed in the correct positions, but the image is is sliding too far vertically as you move the camera. You can verify this by only moving the camera left and right... the target tracks correctly. But if you move it up and down, the image moves too far and leaves the axes lines behind.

The good news is that I believe the tracking is working correctly, and it's only a display problem. Again, you can get a consistent preview if you lock the screen in portrait mode.

I've passed this problem along to the developer who did much of the video/tracking integration.

[Windwoes]

@gearsincorg but, if you lock the phone in portrait and then put the phone in the landscape position, then wouldn't the angle and displacement data from Vuforia need to be adjusted to account for the 90 degree rotation? I'm planning to just modify the UI so that the camera preview fills the entire screen and all the text is just overlayed on top.

Good to hear that's it's just a visual glitch, although I will say that it makes it impossible to tell how well you're tracking the target if the axis drifts off the target, lol.

[lizlooney]

When you create the vuforia parameters, are you setting fillCameraMonitorViewParent to true?

[gearsincorg]

if you lock the phone in portrait and then put the phone in the landscape position, then wouldn't the angle and displacement data from Vuforia need to be adjusted to account for the 90 degree rotation?

Yes, but it already rotated is in the sample.

I ran the camera on it's side, (locked in portrait), for velocity vortex, and the axes responded as expected for target tracking. My experience is that you will get the desired results if the phone orientation you defined in your opmode, matches the phone orientation on the robot (with no auto rotation).

In the samples, the camera "orientation" includes a rotation to lay the camera on it's side.

Whatever you do, it's always wise to do a sanity check to ensure that the axes are responding as expected. I always do this sanity check each time I move/re orient the camera.

[AlecHub]

1. The Vuforia preview is displayed on the Robot Controller, so any modifications of the GUI would be done there, which is permitted.

Hi Phil, fyi \<RS9> is a new rule for the Rover Ruckus season. Beginning this season, teams may not modify the Robot Controller SDK in any fashion.

Modifications to the RC GUI can be viewed as a modification of the Robot Controller SDK.

Because of the new \<RS9> rule, and because nowhere in the rules does it say modifications to the RC GUI is permitted, I suggest that teams get any modification to the RC approved by the GDC, via the Game Q&A Forum, to avoid issues at Robot/Field inspections, and to avoid issues with the Head Ref during matches.

[gearsincorg]

Wow. How did I miss that one. I’m surprised. I’ll need to re-read in detail to clarify my thinking.

-- Phil Malone GEARS Inc.

phil.malone@mr-phil.com Office: 301.387.2331 Mobile: 301.501.7424

[shelbyRobots]

But what does modifying the Robot Controller SDK mean under ? It obviously doesn't mean that you can't touch anything delivered as part of the SDK zip from github - TeamCode folder is part of the zip, but we have to add to it. Likewise, gradle files seem like fair game - we have to modify those to add libs, change compile versions, etc. I would lean towards not being able to modify anything that is delivered as part of the .aar files. But we obviously need a judgement from those on high. Note that the layout can be changed completely programmatically inside TeamCode - without touching the layout xml file. We really need the Q&A forums to be opened - we are now less than 6 weeks from our first qualifier.

[AlecHub]

We really need the Q&A forums to be opened - we are now less than 6 weeks from our first qualifier.

Before they can open the Q&A Forum to questions, the GDC has to pick a Hollywood movie that has a similar theme to the season's game. Once they've reached consensus on the movie, each GDC member has to pick a character from the movie and then submit a request to IT to change their forum login ID to the name of the character they've picked. After IT fulfills the request to change the login ID, the GDC members have to change their avatars. Then and only then can they start taking questions. It is an exhausting process that can take months, but they usually manage to pull it off within a couple of weeks after kickoff. You may be able to help them speed things up a bit by suggesting a movie for this season. Last season they picked Indiana Jones. My suggestion for this season would be Red Planet.

[gearsincorg]

As I read [RS9], it seems totally unachievable and unenforceable.

As others have said... you can't develop with Android Studio if it's not permitted to modify the RC SDK "in any fashion", so clearly that needs to be clarified.

Teams need to know what parts are "Meant" to be modified, and which parts MUST "remain unchanged" for fairness and the ability for FTAs and CSA to debug problems.

Unfortunately there is no real good FIRST example to fall back on. True software "Inspection" has been almost non existent for the 13 Years I've been doing FRC and the 11 years I've been doing FTC, for the very reason that it's very hard to do much beyond just checking version numbers. Modifying the code within the constraints of the SDK/Template has always been required, and inspectors are not in a position to review code.

[AlecHub]

I believe that in previous seasons teams have been decompiling [parts of] the SDK, modifying it, and then building the Robot Controller app from the modified SDK source. Perhaps this is what \<RS9> is meant to stop. If that is the case, I suppose inspectors will have the means of detecting this type of modification.

In any event, "in any fashion" is very troubling. As things stand your robot can fail inspection based on whatever the inspectors and head refs perceive as a modification, which can vary from event to event.

[magneticflux-]

@gearsincorg The maintainers of modular-ftc and, I believe, OpenFTC came to the same conclusion. It's troubling to me that they would attempt to completely lock down their software in the name of "fairness" and "debugging". If a team has skilled enough members to recompile and modify the SDK, it's likely that they are already a well-performing team and it's unlikely that they would resort to asking FTAs for debugging help.

[gearsincorg]

@magneticflux- Don't panic, Once the Tech Team re-read this rule, a change in wording has been proposed.

As for which teams ask for help.. my experience as a CSA is that teams from both ends of the spectrum want answers at competitions.

Novice teams need help with basic functionality. Sophisticated teams often push the limits, and when robot behavior at competition is different from robot behavior at home they want answers as to why.

This is when the FTA and CSA need to fall back on their knowledge base, and correlate their live observations with their known list of behaviors. If internal fundamentals have been changed drastically, then it's hard to provide useful input to teams.

eg: when some teams were streaming video at worlds, they wondered why their performance was suffering. It hadn't been a factor at smaller venues. It's possible in this case that multiple streamers were effecting their own performance as well as other's. Knowing that this is happening, recognizing the consequences, and taking action in the heat of competition is made very difficult if there are no bounds on what teams can do.

There is no knowledge base to fall back on.

[Windwoes]

@gearsincorg your point is well taken.

An alternate solution would be rather than restricting what teams are able to do, tell them that if they modify the internals, they will forfeit help from a CSA/FTA.

[AlecHub]

They changed the rule as follows in the latest release of the game manual:

\<RS09> Software Modification - Teams are not allowed to modify the FIRST Tech Challenge Driver Station in any fashion. When using the Robot Controller SDK, Teams are not allowed to replace or modify the portions of the SDK which are distributed as binary .AAR files.

[qwertychouskie]

Does this mean that OpenRC's stock mode is legal again?

[Windwoes]

@qwertychouskie Yes. Hopefully we'll merge SDK v4.1 into OpenRC "soon-ish".

[cmacfarl]

@qwertychouskie That question can only be answered on the official forums.

@FROGbots-4634 That yes is a tad early. The rule reads "not allowed to replace". A library built outside of the official SDK could be considered a replacement, even if identical. I'd advise posting the question in the forums and waiting for an official answer.

[Windwoes]

@cmacfarl we build the APK using the original, untouched AARs in Stock Mode, though.

[cmacfarl]

I see I just went over and ready your legality page. I thought you were building stock libraries from unmodified sources. But that doesn't seem to be the case. Seems to keep with the spirit of the rule, but I am not a GDC member. I'd still suggest a question on the forums before showing up at a competition with an OpenRC build of your RC.

[Windwoes]

Fixed in v5.5