Investigate and resolve CAN issues

[kammce]

It has been observed that CAN communication is intermittent and unreliable. At times it will work for a few seconds, at times it will work for a minute. But eventually the bus goes down, the MicroMod goes BUS-OFF and the bus signals look chaotic and noisy. The drive systems can no longer control the motors nor get feedback. This has stalled testing on the rover prior to SAR.

Much of the code has not changed but suddenly the CAN stopped working a few days ago during testing by @Coreyboy1820.

Attaching the sn65hvd230.pdf datasheet because I've been having trouble downloading it from other sources.

What we have tried thus far:

• The chips have been replaced 3 times on the Dorito board
• New bodge wires have been installed, in case the previous ones wires were damaged during testing
• A debugger was used to step through to the code to ensure that the issues were not within the code logic but due to missing responses from the CAN BUS.

Looking through the datasheet I have a few ideas of what can be done to improve CAN reliability.

• [ ] Remove 10k (R7) on the Dorito which connects the CAN transceiver's RS pin to ground. This puts the device in slope controlled mode, NOT high speed mode. Although the WaveShare CAN transceiver also uses this pin, maybe with further loading on the bus due to 6 devices, has resulted in a slew that distorts the transmitted signals.
• [ ] Add a bulk capacitor for the transceiver to ensure that the device does not brown out during operation. Will try something quite large for this application such as a 47uF capacitor as shown in this image.

• [ ] Add bus filtering capacitors to smooth out the signals on the bus and absorb any reflections. The size of these capacitors will be in the pF range as to not cause a similar slope/slew rate issue as the slope controlled mode.
• [x] If all of the above fail, we can also consider using split termination to help with filtering.

The list above is in order in which changes will be made. They will be stacked on, one by one until the end. All but the last item seem reasonable and achievable.

[kammce]

After waking up this morning, I've got a few other idea that can be implemented at any point:

• [ ] Remove the 120ohm resistor next to CAN transciever on the dorito
• [ ] Attempt to move towards a single ended topology vs a star configuration as we have currently. See discussion 1

Discussion 1: Star configuration

As stated in this forum https://www.chiefdelphi.com/t/using-can-in-a-star-configuration/139832/2, star configurations are not good for CAN due to reflection issues, and thus, we may want to CUT the CAN traces of the Dorito and rewire them to be a single line terminated by a resistor. This may be extremely difficult to achieve with the overall topology of the 3 wheels rover and the current board designs, but we can at least try to make it less star like.

[kammce]

Discussion 1: Continued

This may work for our use case. We daisy chain the dorito connectors as specified here: https://www.chiefdelphi.com/t/using-can-in-a-star-configuration/139832/10.

As mentioned in this stack overflow discussion and the accompanying TI article we can have stub (unterminated lines) so long as they are short compared the rest of the chain. https://electronics.stackexchange.com/questions/405380/how-far-can-a-terminating-resistor-be-from-the-literal-end-of-a-can-bus. They must be below 1/3 the overall length of the chain. The distance between the slip ring servos and the servo chair is quite small in comparison to the line lengths else where so the hope is that the length is not significant enough to cause reflection issues.

The topology and connections would look like this on the schematic and be made in the layout

[kammce]

Okay, I'm convinced that the true solution is here is to rework the current Dorito board to change the CAN topology from STAR to SINGLE ENDED. All other changes should be held off on until this change has been made. Other changes can be stacked on the new topology to tune this. Here is the plan:

Rework steps

Step 1: Sever all devices from CAN except LEG B

Green mark indicates where the cut will be made on the highlighed (whitish marks) CANH and CANL lines.

Step 2: Connect Steering Servo return path to TXP & TXN

The idea is to use the unused signals on the USB-C connectors to be the return lines for the can signal to continue the daisy chain. The extra servo CANH/CANL connectors will be connected to the servo and the ends soldered to TXP and TXN respectively (CANH -> TXP, and CANL -> TXN).

Image to show that the Taco board does not use the TXN and TXP lines.

Step 3: Daisy chain LEG B CAN to LEG C

The signals from the TXN and TXP lines will show up on the other side of the USBC cable on RXN and RXP. We can connect those lines to the the D+ and D- lines of LEG-C's USB-C connector. This will supply CAN to LEG-C.

Step 4: Repeat step 2 for Servo backpack on LEG C

Step 5: Repeat step 3 for LEG A bug from LEG C keeping the daisy chain

Step 6: Turn off all servo terminating resistors except for LEG A's steering servo

All servos must have their terminating resistors turned off to not distort the bus signals. LEG A being the end of the daisy chain means its the perfect candidate for termination.

New topology diagram

Previous topology

Bonus steps: Cutting as many antenna traces as possible

Every trace that goes nowhere can act as an antenna and a point where signals can be reflected and may need to be CUT OUT to improve signal performance.

[kammce]

Also side note

this could explain why we need to full power cycle the board sometimes when testing to get CAN to work. In the future we may be able to either power cycle the whole can transceiver or use the Rs pin to put the device in and out of low power/standby mode.

[kammce]

This has been resolved as 2 days ago. Had to switch to the new golden slip rings. Will give more details after SAR.

[kammce]

The critical source of the CANBUS issues was copper and tin dust accumulating on the solder joints of the brushes on the brushes slip ring.

This dust comes from the traces slip ring with tin coating.

The copper and tin dust connected the the traces togethers causing a small resistance between contacts.

Brush 1: 36 ohms from CANH to VCC (no other shorts) Brush 2: 2.6kOhm from CANH to VCC Brush 3: 60 Ohms from CANL to GND (discord say VCC, but this was incorrect)

The black ENIG coated slip rings were swapped in and the brushes were cleaned up off with a brush and we observed two things:

1. Short between any signals disappear.
2. Iconic sound of scrapping is gone (its silent now)
3. CANBUS issues went away and signal integrity was restored.

Below is the correct signals:

The following below was the signals with the uncleaned brushes:

[kammce]

One of the servos was broken but the LED lit up. It did not respond to commands.

I removed the can transceiver and wired in a working can transciever and it worked. So we just need to replace these and we can potentially fix many of our motors.

See this video for the motor working: https://youtu.be/4jsvD7vrM9k