Version 1.5 installs a Demon from the Netherworld in your HiTechnic Servo Controllers

[dirthsj]

Thursday we installed version 1.5 in preparation for the competition today (Saturday, 2-06). We didn't test it because all we did was pull the update to our github repo and deploy it - what could go wrong? We didn't change anything, only FTC did.

Arrive at the competition, start running our first match. We have no control over our servos, and they will not stay in one place. We get penalized for problems related to our servo malfunction. Luckily, we have 9 matches and lunch to fix this. It's still not fixed, despite 3 hours of continuous debugging. We talk to another team, and they tell us that version 1.5 has caused problems for them too, and it took them two days to fix all the issues.

We decide that, since 1.5 is the issue, we can use 1.25 which we still have on our laptop. Meanwhile, we miss 2 more matches because we don't have control of our robot. We talk to the local FTAs and they help us get the driver station updated from the .apk file. (They were Very Helpful)

Thinking the problem is finally resolved, we queue up for our 4th match. The problems persist, despite completely uninstalling 1.5. We're confused, so we check the "About" on the app, thinking we must have used the wrong file. It says 1.25. We check the driver station as well. Also 1.25. Highly discouraged, we write a simple program that only moves a servo, using none of our classes. The problem still occurs with that running.

Our 5th and final qualifying match time arrives. We run as a pushbot, and miraculously reach the 17th position in our league.

Reproduction Steps: 1) Create a robot with 3 motor controllers and 1 servo controller (All HiTechnic, connected to a single Legacy Module) 2) Connect 2 servos to any ports on the servo controller 3) Try to set the position of the servos 4) Watch as the servos go to every position except where you set them to, even going backwards. 5) Uninstall the app 6) Install version 1.25 with the same program 7) Watch as the problem DOES NOT GO AWAY 8) Tear out hair in frustration and blame problem on demons

[raunakdoesdev]

Hi,

We are having the same problem with the motor controllers.

We checked all the connection. BTW.

Thanks,

RobotechsFTC

[dirthsj]

Strangely, our motor controllers worked perfectly fine. To clarify, are you using the HiTechnic motor controllers through the legacy module, or are you using the USB motor controllers?

I should mention we are using the standard HiTechnic 180 degree servos, 4 Andymark NeveRest 60s, and 3 Tetrix motors.

[calebsander]

We have been running the beta for over a month using 1 HiTechnic servo controller, and it seems to be working fine.

[Ace314159]

We have 1 HiTechnic Servo Controller and 2 HiTechnic Motor Controllers, and both work fine for us after this update.

[dirthsj]

Hmm... You are welcome to look through our code, if you like. However, before the update our servos were fine, afterwards they were not. Is there any additional information we could provide?

[magneticflux-]

@KingofGamesYami I was looking through your code, and it seems like you use a custom implementation of the LinearOpMode, with method such as "safeSleep", and setting values to be applied every iteration of the loop. We use LinearOpModes for everything, even Teleop. It also worries me that you use a "CustomServo" implementation that could cause issues. This is our teleop and our similar smooth servo system here. We have not updated yet, so we don't know if our system works or not.

[dirthsj]

@magneticflux- The first thing we thought of was CustomServo causing issues. So we commented out the usage of that particular class and tested again - still no change. We use OpMode for teleop, and we are unable to control the servos in that mode at all. Here's what we used to narrow the issue down to 1.5 (by not using any of our classes):

public class ServoTest extends OpMode
  private ElapsedTime time;
  private Servo servo;

  @Override
  public void init(){
    servo = hardwareMap.servo.get( "climber" ) // climber is a 180 degree servo
  }
  @Override
  public void start(){
    time = new ElapsedTime( ElapsedTime.Resolution.MILLISECONDS );
  }
  @Override
  public void loop(){
    if( time.time() > 1000 ){
      servo.setPosition( 0.0D );
   }else{
      servo.setPosition( 1.0D );
  }
}

[magneticflux-]

@KingofGamesYami What exactly happens when you run the code you just provided? Expected vs actual results.

[dirthsj]

Expected: Servo to move to position 1, wait 1 second, move servo to position 0. Result: Servo moves to (seemingly) random positions, moving in either direction, mostly between 0.3 and 0.7.

I've found the video recording of one of our matches, in which you can see one of our servos malfunctioning. https://youtu.be/SbC_QQa77jM?t=19021 Our robot is 4160, on the blue team. The servo that (quite obviously) malfunctions is attached to a simple arm at the front of our robot, you can see it rotating out in front even though the servo is set to 0.0D the entire match (The references to this servo were commented out in our code during this particular match). The other servo didn't work either, but it's harder to see.

It's especially easy to see the problem at 5:17:13, where the servo looks like it's somewhere around 0.4

[Ace314159]

Actually, now that I think about it more. My team has also experienced this problem. We fixed it in our TeleOp by setting a starting position and a position if none of the buttons that actually move the servo itself. We set our continuous servos to 0.5 and that stopped them from spinning.

[gearsincorg]

First let me say what everyone is thinking. You did WHAT?

You downloaded a bunch of unknown code into your development environment. Built and deployed it. Then you downloaded a new app into your driver station.

And then you....., didn't even run it between Thursday and Saturday when you went to competition.

You don't even mention running the program during software or field inspection???

Given that you say that you deleted everything AND re-loaded your old code and it's still broken, it's completely clear to me what happened.

Someone gave your kids the task of cleaning up the wiring, or repairing something, and in the process they switched the Servo wires to the incorrect ports. (or someone switched the servo allocations in the phone's configuration file). It completely explains the symptoms you are seeing, and it's extremely common.

So my suggestion would be that you check that the servos are wired correctly from the servo to the Servo Controller, that the plugs have the correct polarity, and that the config file has the servos on the matching ports.

Phil.

[magneticflux-]

@gearsincorg Seeing as other teams have also has issues, I doubt several teams simultaneously mis-wired their servos. I also expect a degree of reliability with an official update, so while you're remarks are true, they seem a bit too severe. @athenix and @Ace314159 could you verify that you indeed had the same problem?

[wernerlin]

Is it possible that your issue is hardware-related and that the timing with the update is coincidental? We have had intermittent issues with components going haywire as you described - motors just taking off (once launching our bot off a table), servos flipping about, etc. In each case, we had to swap out / fully recharge the battery and do the turn-off-and-wait-30-seconds procedure before restarting and trying again. In all our cases, we hadn't changed any code or installed any software updates.

[dirthsj]

It's possible, but other teams had the issue with the update as well. The only thing we didn't try (IIRC) is swapping out the servo controller, mostly because we lacked the time to try everything.

[magneticflux-]

@wernerlin I've never seen anything as violent as what you describe, but on several occasions our robot's motors have failed to move the first time they are set and servos temporarily initialize to an incorrect position. @KingofGamesYami Have you completely removed all traces of the update from your code and phone? If you have, the only idea I have is that the update changed how the servos are commanded and somehow corrupted the firmware on the controller through incorrect data packets. If you use git integration in Android Studio (which is pretty great btw), you can check out an earlier version and test that by viewing the commit log tab.

[dirthsj]

@magneticflux- I believe we did, yes. I went to apps, then long-pressed to use the ZTE speed's uninstall feature. We do use git integration in Android Studio (I agree, it is pretty great), but it was/is kind of broken after I added an mp3 file to raw (Android Studio wouldn't upload it for some reason or another, so I switched to github desktop to get it to upload. Which accidentally uploaded a bunch of stuff like local.properties). Because of this glitch, I re-cloned the repository to a different directory. That left a version still on 1.25 that was relatively simple to revert to (this is what we used). After that, I simply copy/pasted the updated _4160 directory to that earlier repository.

[gearsincorg]

Ace314159’s issues appear to be the standard problem when teams don’t initialize ALL their servos in the start of the program eg: init() It’s a surprise to many first time servo users that when you output to one server, all the others move to their default locations. This is nothing new in the latest release. It’s always been this way (even with the legacy servos).

My remarks may cause some discomfort, but if it prevents people from running untested code in competition, then I think it’s worth the pain.

Also remember that I was posting to a thread entitled: Version 1.5 installs a Demon from the Netherworld in your HiTechnic Servo Controllers I considered this “a bit harsh”.

My money is still on swapped wires.

Phil.

Note: My original post was meant to be read in a nurturing and caring tone of voice. Not harsh :)

@gearsincorg Seeing as other teams have also has issues, I doubt several teams simultaneously mis-wired their servos. I also expect a degree of reliability with an official update, so while you're remarks are true, they seem a bit too severe. @athenix and @Ace314159 could you verify that you indeed had the same problem? — Reply to this email directly or view it on GitHub.

[coder-bot]

Definitely check the hardware – my team has seen servos act up just as you've described, and when they are replaced, they work just as programmed.

On Monday, February 8, 2016, wernerlin notifications@github.com wrote:

Is it possible that your issue is hardware-related and that the timing with the update is coincidental? We have had intermittent issues with components going haywire as you described - motors just taking off (once launching our bot off a table), servos flipping about, etc. In each case, we had to swap out / fully recharge the battery and do the turn-off-and-wait-30-seconds procedure before restarting and trying again. In all our cases, we hadn't changed any code or installed any software updates.

— Reply to this email directly or view it on GitHub https://github.com/ftctechnh/ftc_app/issues/144#issuecomment-181510922.

[dirthsj]

I've just tried replacing the Servo Controller with a different one, and using a completely new (ei unopened until now) servo. Same results as before.

[dirthsj]

OK really weird result now, problem persisted after stopping the opmode and removing the cable from the phone. I removed all connections excepting the servo controller (through legacy module) and servos. Also replaced the cable from the legacy module to the servo controller.

We don't have replacements for the legacy module, CPDM, Phone Cable, or the cable from CPDM to the Legacy module, so I can't test if those are the problem.

My current guess is that something in the Legacy Module was updated incorrectly. Not sure how we could fix / test that, given no replacement.

[dirthsj]

Update: To completely eliminate any questions, I downloaded the ZIP file of 1.5 and added one opmode to it.

public class Test extends OpMode {
  private servo a, b;
  @Override
  public void init(){
    a = hardwareMap.servo.get( "a" );
    b = hardwareMap.servo.get( "b" );
  }
  @Override
  public void loop(){
    if( gamepad1.x ){
      a.setPosition( 1.0D );
    }else{
      a.setPosition( 0.0D );
    }
  }
}

(also added it to FtcOpModeRegister)

Problem persists

I also created a new config file, with only the servos and servo controller in it.

Also tried 3 different ports on the CPDM for the Legacy Module.

[magneticflux-]

@KingofGamesYami That is really weird. I'm surprised that could even happen, given the number of safeguards against runaway robots in the code. Perhaps something is wrong with the connection to the servo. Do you use extension cables for the servos?

[dirthsj]

@magneticflux- We used extension cables for one servo, but we immediately removed all of them when we began testing things.

[gearsincorg]

Time to do some basic hardware sanity checks.

1) Is the RED power LED on SOLID, on the Legacy Servo Controller?
(It needs it own dedicated 12V power)

2) When you start the program and enable the servos for the first time, do the 180 Deg servos hold their position, or are they limp? Limp servos are not being driven, and will respond randomly to static etc.

3) Are you using analog or digital servos? Analog servos will go limp when they lose the PWM signal. Digital ones wont. Digital CR servos will keep spinning after the PWM is shut off.

Be as specific as possible. This isn't a software problem. It's hardware.

Note: Servo Controllers and Legacy controllers aren't ever "updated". They have fixed firmware. They cant be effected permanently by changing the app code on the phone.

[dirthsj]

@gearsincorg 1) Yes (We have nothing else currently plugged into the CPDM)

2) in init() they are limp. When we start loop() they are always set to a position (see our test code above) The problems do not occur until loop() starts.

3) We are using HS-475HB HiTechnic Servos. I'm not sure if those are analog or digital, but it's not continuous rotation.

I'll be on [redacted] if you wish to talk about anything directly.

[magneticflux-]

@KingofGamesYami Would it be possible for you to create an Autonomous using the LinearOpMode class and set the servo's position both before waitForStart() and after? It seems like every team that has had problems has been extending the OpMode class. An example of a LinearOpMode autonomous is here.

[dirthsj]

@magneticflux- our robot suddenly started working again while using the 'test' opmode, so we reinstalled our old project (eg the one I linked above). It now works perfectly. I have no idea what we did to fix it, but we are now unable to replicate this issue now. Closing, as it's clearly not an issue now. Next year we will be able to run our robot.

[magneticflux-]

@KingofGamesYami I'm glad to hear that the issue has gone away for now. My team will deliberate considerably before we update our repository. Good luck to you next season!

[BillPedersen]

I have let this play out as I thought it would be interesting to see where things landed. I am a mentor of a couple teams and the director of an affiliate partner. I also have about 40+ years systems programming and consulting.

The issue I have seen here equates to the desire to incorporate the "latest" and "greatest" release of a framework into a production environment without testing. This is something which should NEVER be done!

I know we are working with new technology and new environments and even new, recently learned skills, but the key here, and it is something ALL TEAMS need to learn is: DO NOT INTRODUCE NEW SOFTWARE THE WEEK OF COMPETITION!

I realize there is a temptation to use the newest framework as soon as possible because it is NEW. It might even have a solution to a problem you are having. But, it might also KILL your environment due to untested paths YOUR code uses.

The ideas I am presenting here are those tried and true principles utilized by mainstream business to keep their Business Critical Applications running. Your business critical application is YOUR robot! You do not want to disrupt its operation the week of a championship! Do not do updates. Carefully review all YOUR changes. Make sure YOU OWN the issues and are not introducing something you have no control over.

Good luck folks!

Bill.

[rgatkinson]

FWIW: I took the latest SDK bits and ran the following opmode (very similar to the one mentioned above, but not quite identical).

public class ServoTest extends OpMode
    {
    private Servo a;

    @Override
    public void init()
        {
        a = hardwareMap.servo.get("servo");
        }

    @Override
    public void loop()
        {
        if (gamepad1.x)
            {
            a.setPosition(1.0D);
            }
        else
            {
            a.setPosition(0.0D);
            }
        }
    }

It worked as expected; I was unable to reproduce the problem. Nor have I seen any of the described symptoms of such in somewhat rather non-trivial testing along similar lines. Based on this, and considering the above mentioned points and failure scenarios described by others, my strong suspicion is that hardware issues were the underlying source of the regrettable problems here, not the SDK update.

[magneticflux-]

@rgatkinson Good to know, thank you! We'll be updating soon, since we have several weeks until the Missouri state competition to fix any issues.

[dirthsj]

@BillPedersen Are we not required to update per rule RS03?

RS03 Java is the recommended programming language for the FIRST Tech Challenge. Programming must be done using one of the following applications: a. Android Studio – a text-based integrated development environment. b. App Inventor – a visual blocks-based programming tool. c. Java Native Interface (JNI) & Android Native Development Kit (NDK) – teams can incorporate native code libraries into their apps using the JNI framework and the Android NDK. If updates are announced by FIRST later in the season, Teams must install them prior to the time of competition.

[BillPedersen]

@KingofGamesYami

I would recommend submitting that question the FIRST Tech Challenge Forum.

The issue I see is you have a conflict here. You want to be able to compete and you also want to follow the rule. On a strict reading of the rule it would suggest if there as a competition on the day after the announcement of a release of the a new version you would not qualify if you had not installed the new version. That really is not practical. And so you need to then consider is 5 days sufficient time to get the new version installed and tested in your environment? Generally I would argue no, it is not sufficient time.

You have to balance testing and reliability versus rule compliance.

You also have to consider that most teams are still learning this technology.

So to expect teams to have the newest software framework, released the week of a Championship, to be running that new framework is impractical.

I realize I am not the authoritative answer you would like, but I also know what is reasonable given my experience.

Bill.

[xyzzycoder]

Bill,

While I think that your suggestions are, strictly speaking, correct (if not currently a violation of the rules), I think that you are assuming far-deeper level of understanding of software than most teams have access to.

The level of complexity of the new platform is huge. Ridiculously huge. When I hire Java developers as a manager I ~expect~ that they will spend 1-2 days just setting up their programming environment. Putting this kind of barrier in front of kids (and mentors) who frequently know little or nothing about programming does not create positive experiences. I do not think that FTC could have picked a more heavy-weight and burdensome platform for the kids to learn to program. Java takes the cake.

Most of the kids that I have been around did not, until very recently, even understand the concept of a debugger - much less the nuances of source control with Git. Most mentors probably do not know what change tracking is, how it is to be done, etc.

Most mentors probably could not even understand what you’re writing. There is a difference between knowing what you don’t know and not knowing what you don’t know. I think , with the myriad of changes this year, most have been put in a situation where they don’t know what they don’t know.

Aaron

On Feb 8, 2016, at 4:51 PM, BillPedersen notifications@github.com wrote:

I have let this play out as I thought it would be interesting to see where things landed. I am a mentor of a couple teams and the director of an affiliate partner. I also have about 40+ years systems programming and consulting.

The issue I have seen here equates to the desire to incorporate the "latest" and "greatest" release of a framework into a production environment without testing. This is something which should NEVER be done!

I know we are working with new technology and new environments and even new, recently learned skills, but the key here, and it is something ALL TEAMS need to learn is: DO NOT INTRODUCE NEW SOFTWARE THE WEEK OF COMPETITION!

I realize there is a temptation to use the newest framework as soon as possible because it is NEW. It might even have a solution to a problem you are having. But, it might also KILL your environment due to untested paths YOUR code uses.

The ideas I am presenting here are those tried and true principles utilized by mainstream business to keep their Business Critical Applications running. Your business critical application is YOUR robot! You do not want to disrupt its operation the week of a championship! Do not do updates. Carefully review all YOUR changes. Make sure YOU OWN the issues and are not introducing something you have no control over.

Good luck folks!

Bill.

— Reply to this email directly or view it on GitHub https://github.com/ftctechnh/ftc_app/issues/144#issuecomment-181666101.

[BillPedersen]

I fully agree that this has been a HUGE mountain for many to climb. But I also believe they are moving forward with it and will only get better in the coming seasons.

Better to start talking about it and then give the teams something to look forward to in my opinion.

We also need to work with improving our mentors' skills as well as recruiting more mentors with more current and targeted skills.

I see the need in many of our teams here in South Carolina - an interesting challenge for me and them.

Bill.

[RobJLHollis]

Aaron:

Thank you, it is nice to see someone else looking out for those not on the bleeding edge. In our region, the first two competitions had a total of 2 teams who had an autonomous program. Both bots, rolled forward a couple revolutions and stopped. All of the rest simply watched their controllers, the ceiling, the floor, as their robots stood along the wall like many bashful young men and women at their first school dance. This is a litmus test showing that Java via either MIT App Inventor or via Android Studio by itself is a huge hurdle, and based on the number of teams who were able to run it, you can see how many high schools have the technical expertise to navigate the endless sources of information about them.

I could overlook this if teams didn’t have complex autonomous programs, but for the vast majority to have nothing, shows the barrier to entry moving to Java has had.

I myself have android studio setup (I think, but I don’t know what I don’t know) and none of my gradle projects are making it to my phones to even test. I can see the debugger, I can use git (at a very rudimentary level), was able to pull the basic program into the studio. But I have errors when trying to compile, and even on opening of the studio.

I think what is needed is a step by step how-to, for installing and configuring the environment, ending with the basic K9teleop program (compiled from the environment) running on a phone hooked to a bot. Without wheels on the ground, in an environment where the wheels themselves have changed, many teams are experiencing frustration that is not positive for the program.

The biggest mistake this year was moving to this, saying how easy it was to implement, pointing to the millions of users companies and devices, and telling people to go sift through all the available sources to find the one that works with the components we are using.

If you know where that type of walk-though is, that is something I don’t know I don’t know. Please reply with any direction.

I appreciate again your voicing of the point regarding overall configuration complexity.

I am a programmer by trade, Visual studio, and write and configure systems to talk to each other for over 20 years. I have double undergraduate degrees in Management of Info System and Computer Science, Business Application Programming, and a Masters in IT Project Management. All of my education simply shows me there is more to learn, and if I’m not sure where to start, my heart goes out to any teams staring, and ESPECIALLY any teams who have formed in the past 2 years, and thought they just got the hang of the last system (Labview and RobotC).

Thank You,

Rob Hollis

mailto:RobJLHollis@Hotmail.com RobJLHollis@Hotmail.com

Primary: 650 7EDI NOW | 650.733.4669 PDT

https://www.linkedin.com/profile/public-profile-settings?trk=prof-edit-edit-public_profile https://www.facebook.com/robjlhollis https://twitter.com/RobJLHollis http://cedarcrestftc.blogspot.com/

From: Aaron Morse [mailto:notifications@github.com] Sent: Tuesday, February 09, 2016 2:30 PM To: ftctechnh/ftc_app ftc_app@noreply.github.com Subject: Re: [ftc_app] Version 1.5 installs a Demon from the Netherworld in your HiTechnic Servo Controllers (#144)

Bill,

While I think that your suggestions are, strictly speaking, correct (if not currently a violation of the rules), I think that you are assuming far-deeper level of understanding of software than most teams have access to.

The level of complexity of the new platform is huge. Ridiculously huge. When I hire Java developers as a manager I ~expect~ that they will spend 1-2 days just setting up their programming environment. Putting this kind of barrier in front of kids (and mentors) who frequently know little or nothing about programming does not create positive experiences. I do not think that FTC could have picked a more heavy-weight and burdensome platform for the kids to learn to program. Java takes the cake.

Most of the kids that I have been around did not, until very recently, even understand the concept of a debugger - much less the nuances of source control with Git. Most mentors probably do not know what change tracking is, how it is to be done, etc.

Most mentors probably could not even understand what you’re writing. There is a difference between knowing what you don’t know and not knowing what you don’t know. I think , with the myriad of changes this year, most have been put in a situation where they don’t know what they don’t know.

Aaron

On Feb 8, 2016, at 4:51 PM, BillPedersen <notifications@github.com mailto:notifications@github.com > wrote:

I have let this play out as I thought it would be interesting to see where things landed. I am a mentor of a couple teams and the director of an affiliate partner. I also have about 40+ years systems programming and consulting.

The issue I have seen here equates to the desire to incorporate the "latest" and "greatest" release of a framework into a production environment without testing. This is something which should NEVER be done!

I know we are working with new technology and new environments and even new, recently learned skills, but the key here, and it is something ALL TEAMS need to learn is: DO NOT INTRODUCE NEW SOFTWARE THE WEEK OF COMPETITION!

I realize there is a temptation to use the newest framework as soon as possible because it is NEW. It might even have a solution to a problem you are having. But, it might also KILL your environment due to untested paths YOUR code uses.

The ideas I am presenting here are those tried and true principles utilized by mainstream business to keep their Business Critical Applications running. Your business critical application is YOUR robot! You do not want to disrupt its operation the week of a championship! Do not do updates. Carefully review all YOUR changes. Make sure YOU OWN the issues and are not introducing something you have no control over.

Good luck folks!

Bill.

— Reply to this email directly or view it on GitHub https://github.com/ftctechnh/ftc_app/issues/144#issuecomment-181666101.

— Reply to this email directly or view it on GitHub https://github.com/ftctechnh/ftc_app/issues/144#issuecomment-182107982 . https://github.com/notifications/beacon/AP9hkT3_3X20mgWLh9OxK7VgkF84VCydks5pil_tgaJpZM4HU_Fb.gif