Phoenix Pro Position PID has not been functional since firmware v23.6.4.1

[mjansen4857]

We have tried updating the pro firmware on our falcons with every new release since 23.6.4.1 and the PositionVoltage control mode has not been working properly in every version since then. The system will heavily oscillate in place while not really moving toward the target. I believe it is oscillating between full forward and full reverse but I am not sure. We haven't been able to do much testing with it since it could be damaging to the robot.

Downgrading the firmware to 23.6.3.2 fixes the issue and the system works as expected.

[CoryNessCTR]

I'm not seeing any issues with current latest (23.6.5.0), using this repo as my testbed: https://github.com/CrossTheRoadElec/SwerveDriveExample

The only change I see in 23.6.4.1 is a fix to setting position while the motor is inverted. Are you doing that anywhere in your code? Could you link to your code's repo and I may be able to look more into it?

Keep in mind we fixed kD in 23.6.3.2 so that it's properly applied. Could the issue be derivative oscillation and you're actually downgrading below 23.6.3.2 where the derivative isn't strong enough to produce the oscillation you're normally seeing? https://api.ctr-electronics.com/changelog#20230206

[mjansen4857]

Our repo is not currently public so I can’t link to it. But I can send relevant snippets in a bit.

We are definitely downgrading to 23.6.3.2. Was kD broken recently before it was fixed or had that been an issue for a while? Could the change log have that fix listed for the wrong version somehow? We can try playing with the derivative term later today. It has worked the whole season up to this point and we noticed it working while tuning it so we assumed it to be an issue with firmware.

[CoryNessCTR]

23.6.3.2 definitely has the fix for kD, I went back through our git history to confirm this, and it would've been a problem with any firmware before 23.6.3.2.

Snippets would definitely help with something like this, especially since I can't reproduce it on my setup. A minimum example that reproduces the issue would also be helpful if you can easily put one together.

[mjansen4857]

So I can confirm that we were running 23.6.3.2 without seeing this issue. We tried upgrading again tonight and reducing the kD term which helped, however, it is still slightly present even with a near 0 D value. We used to run at kD = 3.0 and now need kD = 0.1 to fix the issue most of the time. It will once in a while have some oscillation for a split second before returning to normal with kD = 0.1. We never had any oscillation at all in 23.6.3.2. We will do some more tuning over the coming days, but may just try to tune for kD = 0 since that has no oscillations.

Relavant code snippets:

Configuration:

    rotationMotor = new TalonFX(rotationMotorCanID, canBus);
    rotationConfig = new TalonFXConfiguration();
    rotationConfig.MotorOutput.Inverted = InvertedValue.CounterClockwise_Positive;
    rotationConfig.Slot0.kP = ROTATION_KP; // kP = 6.0
    rotationConfig.Slot0.kD = ROTATION_KD; // kP = 3.0 pre-update, 0.1 post-update
    rotationMotor.getConfigurator().apply(rotationConfig);

Output:

StatusCode rotationStatus =
        rotationMotor.setControl(
            new PositionVoltage(
                targetAngle / ROTATION_DEGREES_PER_ROTATION,
                useFOC,
                0.0,
                0,
                false));

[CoryNessCTR]

I'm still unable to reproduce this so far. In my case, controlling a swerve azimuth, I have a kP of 10 and a kD of 0.3, and regardless of what firmware version I use (23.6.3.2 or 23.6.5.0) the controller is on the cusp of oscillation. Sometimes when I translate it will oscillate and sometimes when I translate it won't oscillate.

With your initial set of gains, it will always oscillate in my setup. The kD being 3.0 feels high in my experience, I generally keep the kD around 0.2 when doing closed looping on Position with Voltage.

Could you explain the mechanism more? Maybe there's some dynamics in your mechanism I'm not able to reproduce with a swerve azimuth controller. Are you using FOC? I've tried applying with and without FOC without reproduction in any case, but I want to match your setup as close as possible.

Would you be comfortable adding myself or one of the other CTR employees as an outside collaborator to your repo to dig into this more? We treat team's code as confidential, so we won't share it outside our organization.

[mjansen4857]

Did some more testing tonight and we are still getting some weird oscillations randomly even with a 0 kD term. The mechanism is the rotation of an SDS mk4 module we are running the PID loop with the internal encoder as feedback, not the cancoder. We are using FOC. I can look into adding you to the repo in a bit.

[CoryNessCTR]

It sounds like your setup is nearly identical to mine in that case. The only notable difference I can see is I normally run fused CANcoder, but I switched to using the internal sensor for my tests since that's what you used.

Since our mechanisms are nearly identical, could you run the example we have here? https://github.com/CrossTheRoadElec/SwerveDriveExample It also uses the SDS mk4 modules, so the turn gains as-is should be close enough for your robot. If you can at least confirm the azimuth controllers are reacting as you'd expect that's really helpful.

If the example works, that's very telling, and if it doesn't that's also telling.

[mjansen4857]

Yeah we can give that a try tomorrow probably. We would use fused cancoder but remote sensors don’t seem to be supported in sim (motor will never output any voltage if it’s enabled) so we’ve been avoiding it for mechanisms that don’t absolutely need it. Not sure if that’s an issue you’re aware of or not.

[mjansen4857]

So the oscillations we are now seeing occur both in our code and the example project you sent. They also occur regardless of the firmware version we use(at least 23.2+). Only happens when the robot accelerates hard perpendicularly to the current module rotation. We have also tried with 0 kD and this still occurs, but it looks and sounds like oscillation from a too high kD value. We have verified that the robot is not entering brownout protection while this is happening so we are not really sure what’s going on here. While this is happening, the status lights on the CANcoders start blinking red. They are normally in the green. I’m not sure if the oscillation is causing the encoder to think the magnet is out of range, or if the encoder thinking the magnet is out of range is causing the oscillation.

We unfortunately will not be able to provide any more info besides this until sometime this weekend since we will not have the robot until then.

[mjansen4857]

We figured out the issue. Turns out one of our falcon shafts was bent causing some gear slip.