search

CrossTheRoadElec / Phoenix-Releases

Release repository for Phoenix-Framework and Phoenix-Tuner
76 stars 7 forks source link

Python: Can't transmit CAN frame to Talon FX Phoenix 6 #75

Open lospugs opened 4 months ago

lospugs commented 4 months ago

Hello, we're having a fairly consistent (meaning at least once every meeting) where we end up not being able to command our TalonFX motors. It's not so consistent that we can reproduce it reliably (at least haven't figured out what triggers it), but it's definitely a bug that exists that has me worried.

-1001 Could not transmit CAN Frame. talon fx 22 () Set Control DutyCycleOut

So far the only fix that I'm aware of is restarting the robot code from the DriverStation.

This has me a bit worried for competitions.

From my Pip List: phoenix6 24.1.0 robotpy 2024.2.1.1

CoryNessCTR commented 4 months ago

I believe your issue was resolved in a robotpy installer update. https://github.com/robotpy/robotpy-installer/issues/111

Can you update to the latest version of robotpy installer and re-test?

If it was that issue, then the issue only ever occurs when a running program is stopped for another program to run, meaning it will never reproduce when the robot first powers on, so encountering it at competition is very unlikely unless your process is to restart robot code or redeploy robot code before every match without power-cycling the robot.

But updating should also resolve the problem.

Please re-open if you find that updating does not resolve the issue.

lospugs commented 4 months ago

Thanks for the response @CoryNessCTR . Unfortunately, or fortunately I guess, this is running with the latest installer.

It is not related to the deploy issue that was fixed with the last robotpy-installer update.

This happens after driving for a while and then at some point will periodically lose CAN comms with the Talon FX objects. Note, it does not happen with the SRX controllers on the bus so it feels related to Phoenix 6.

lospugs commented 4 months ago

@CoryNessCTR -- This is still happening as of Saturday.

robotpy 2024.2.1.2 robotpy-installer 2024.2.1

lospugs commented 4 months ago

It's only the Talon FX objects, never the SRX.

CoryNessCTR commented 4 months ago

Do you have Driver Station logs of when this happens? Can you start the signal logger and attach a hoot file of when it occurs here?

How long does it typically take to reproduce?

lospugs commented 4 months ago

Do you have Driver Station logs of when this happens? Can you start the signal logger and attach a hoot file of when it occurs here?

How long does it typically take to reproduce?

I will try to get you some metrics and logs tonight at our shop.

lospugs commented 4 months ago

Okay, sorry, I forgot to enable the signal logger so I'm not giving you actionable information to see what might be going on. Based on my observation, and about 2 hours of testing yesterday, I was able to induce the condition 4-5 times total.

We have had some issues with can wiring, where collisions and bumps would cause the can bus to lose devices.

The only thing I noticed across the 4-5 times we saw the issue was that in DriverStation on the information printout I would see 2-3 errors of TX Full. Re-starting the robot code fixes the CAN stalemate and we can keep driving.

However, it seems correlated with the error fallback from the condition that populates driversation's TX Full error message. The code doesn't recover from that state without a restart.

CoryNessCTR commented 4 months ago

I'm starting to believe this may be a wiring issue more than a library issue. When the issue occurs, are the SRXs controllable? Do you see devices in Tuner? Can you control them there?

We have had some issues with can wiring, where collisions and bumps would cause the can bus to lose devices

This further indicates to me the problem stemming from a CAN wiring issue.

The next time this happens, leave the robot in its "bad" state and reduce the bus to a single device. It should be able to recover. If it does, then I'm even more confident there's a wiring issue and you can investigate the wiring harness further. Check for a short between CANH/L and Ground/Vbat/Chassis or for flaky connectors. It it doesn't, then triple check with some other single device, potentially try a different roboRIO, try different CAN wires.

The signal log may still be useful, the driver station logs I think are useful if you have them. Send either of them over if you get the chance to.

lospugs commented 4 months ago

The CAN devices are all visible on Phoenix Tuner when the error happens.

I didn't think to try to use Tuner to control them, but I'll do that if I can induce the problem tomorrow in testing.

Thanks for the notes, I'll keep you posted.

Edit to add: One thing that was confusing to me with regard to it being a wiring issue is why does it get solved by restarting the robot code? It may very well be a wiring problem where the issue presents and then the shorted (or whatever wiring problem) issue fixes itself coincidentally but it's a guarantee that restarting the robot code (not the roborio) fixes it.

I'm concerned enough about the wiring that I'm having the students re-run all the wiring in the robot because I don't trust it will hold up to competition.....so I'm also of the opinion this very well could be a wiring issue.