Investigate shivering of servos

[nerdoug]

Both HexaFloorRide (HFR) robots show an unexpected behaviour when more than about 3 servos are activated. Some of the activated servos will "shiver" or do a low speed oscillation with small amplitude as if they're in some sort of a feedback loop. The movement can be stopped by dampening the movement, but sometimes starts up again. Also it was observed that while the shivering is happening, the +5 voltage as measured at the PCA9685 is reduced by about .4 volts.

The robot has issues with unwanted "shivering" motion. We are investigating a number of ways to address this issue.

• This ticket, #73, is focused on fixing the electrical circuit to see if that makes the shivering go away.
• This ticket, #172, is about investigating the use of rubber bands (or other methods) to per-load the lags to stop the unwanted movement.
• Ticket #168 is about investigating the use alternate motor options like Boston Dynamics and/or MIT mini cheetah.

[nerdoug]

When I noticed the voltage reduction of about .4 volts, I checked the output of the power supply part way through the rather crude wiring path to the PCA685's green wire terminals. It was 5 volts at my intermediary measurement point (IMP), but about 4.6 at the header pin for V+ on the controllers. These are wired together with no intervening components, and should be the same voltage. The only explanation I can think of is that the wiring path between my IMP and the controllers has some resistance in it.

The wiring from the power supply to the IMP is quite robust, confirmed by the voltage still being 5V. Between the IMP at the controllers, both the +% and the ground path look like this:

IMP ending in a loop of #14 solid wire alligator jumper (which I had to replace earlier this evening due to intermittent resistance/open) about 12 inches long about 24 gauge jumper wire 3 inches long breadboard row 22 gauge wire to each controller about 16 inches long

My next step will be to rework these connections with low resistance wiring and solid interconnections. No breadboard connections, no alligator clips, and the thickest wire I can find. Then I'll test this configuration to see:

• if the shivering is unchanged, reduced, or gone
• if the voltage reduction still happens
• if it does, at what point in the circuit does the drop occur?

Some other thoughts:

• increase the capacitor on the controller boards?
• find the resistance per foot of the various wire gauges we're using & calculate the total
• use an ohmmeter to measure the resistance of various parts of the "current" setup
• see if anyone has documented a similar project on the internet, and examine their wiring
• make up a diagram of the 5V power wiring, with some resistance in both sides, and apply Ohm's law

[nerdoug]

another todo: look at the specs for the servos, and see what the minimum voltage is according to the manufacturer. Nice to know if we're outside the normal operating voltage, or if it should be working even at the lower voltage.

[nerdoug]

issue 8, related to wire size for motor wiring, is related to this one, and has some useful info. for resistance & max current for various wire gauges, you can check this website: http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/wirega.html

[nerdoug]

Before changing anything from previous testing, I measured the resistance of various parts of the power wiring. When I shorted the probes on my ohmmeter, I got a .3 Ohm reading, which probably reflects probe resistance and miscalibration. I've subtracted this from my readings, which are all in Ohms.

From IMP ground to:

• power supply ground: .1
• PCA servo ground pins: 1.5
• mini voltmeter GND lead: 2.6

From IMP +5 to:

• power supply +5: .1
• PCA green terminal .5/.4 on both PCA's
• PCA V+ pin: 29M and changing (suspect meter is effectively charging the capacitor)

From green terminal +5 to V+ pin

• open on one PCA, 12M on the other. blaming this on capacitor as well.

Conclusions:

• my 5V power wiring has problems with excessive resistance
• need to take mini-voltmeter readings with a grain of salt

Next steps:

• see if plugging observed resistance into Ohms law tells us anything
• use oscilloscope to look at 5V levels at various places in current setup
• replace 5V wiring with low resistance path, reassess shivering, and redo resistance measurements

[nerdoug]

Specs for HS-485HB servo are here: https://hitecrcd.com/products/servos/analog/sport-2/hs-485hb/product

and the operating voltage is stated as 4.8 - 6.0 volts. So, our voltage dips are taking us out of the specified range, and it might be expected that the servos aren't working properly.

[nerdoug]

Completed the Ohms law analysis. Having the observed resistance on 5V wiring is catastrophic, and reduces the voltage available to the PCA9685 controllers below their specified requirements. Details in Servo-power-wiring.odg and pdf, which I'll try to attach to this ticket, and upload to the repo.

Servo-power-wiring.pdf

[nerdoug]

If the Ohm's law stuff is correct, then cranking our bench power supply voltage from 5V to 6 volts should solve the problem. Experiment time.

[nerdoug]

Still getting general shivering with voltage feed at 6 v, although it didn't show up until I had about 7 servos active. The mini-VM shows around 5.7 volts at PCA9685, but bench power supply shows current cycling between .4 and .8 amps. Firing up the scope.

[nerdoug]

If I was seeing total current hitting .8 amps, and the 2 controllers were out of sync, then one controller may have been hitting .6 amps. Even with 6 volt input, this means 4.86 v to controllers, which is close to the limit. Think next step, (after replacing the PCA9685 that I smoked) is building a robust wiring scheme for 5V and start clean observations/testing on it

[nerdoug]

Completed rebuild of power wiring, including rebuild of my DIY power supply. Still getting shivering, maybe more. Voltage at PCA9685 green terminal block is rock solid. total current on power supply varies. with 12 servos active, current is generally between .39 and .44 amps. Seems that only the hip servos are shivering?

with all 18 servos at pwm=300, just 2 hips are shivering, and current ranges from .39 to .65

disconnected I2C bus from ESP32 and shivering continued

[nerdoug]

Observation: in main.h, we set the servo frequency, SERVO_FREQ to 50. In the Adafruit docs, https://learn.adafruit.com/16-channel-pwm-servo-driver?view=all their example sets the frequency to 1000. Have also seen examples where it's set to 60. Could experiment with different frequencies. Could also scope the PWM signal on the controller to see if the servo is oscillating on its own, or if it's being told to oscillate

[nerdoug]

changing SERVO_FREQ to 200 caused servo position to be wildly off. Did we go through a tuning process to determine the appropriate value for this parameter which depended on the frequency of a crystal on the PCA9685? If so, we probably should do it again, for all of our controllers, especially considering my tendency to smoke them.

I have reproducible shivering behaviour on leg 4 hip (servo 0 in STP command) and leg 5 hip (servo 3 in STP command)

servo 0: a)position 300 is always a stable move b)position 320 shivers for 1 second, then stabilizes c)position 280 shivers for 1 second then stabilizes

servo 3: d)position 300 is always a stable move e)position 320 shivers, and continues to shiver f)position 280 shivers, and continues to shiver

(Bear in mind these servos haven't been calibrated, and may not be centered)

if you swap which port on the PCA9685 these servos are connected to, the above behaviour follows the servo, not the controller port. This doesn't necessarily mean the servos are the cause of the shivering, as they could be reacting slightly differently to some other problem.

Next step: very carefully (avoiding more smoke release) scope the PCM signal in the above test cases, and see if the command pulses sent to the servos are stable in all cases. Did this, and the control signals are very clean, with the duty cycle pulse at about 3.2 volts

Added extra connections between motor ground and logic ground with no improvement.

Did several rounds of testing to confirm that it's only HIP servos that shiver. The only think I can think of that's unique to them is that the weight on them tries to pull their axis of rotation out of alignment, which might stress the feedback mechanism. The knee and foot bear the weight of attached leg bits more as a rotation around the axis. A bit far fetched, but I can't think what else differentiates them. Should swap cables with a hip and knee, and see if the problem follows the servo, or stays with the hip location. I'll be shocked if it doesn't stay with the hip position. OK, that's the next test, which I may get to later this evening.

[nerdoug]

Thought of a possible test of my "prying the axis" theory. Use a string from "the ceiling" to support the knee servo, removing the prying force on the hip servo. Or possibly support it from below in some way that allows the small amount of movement for test STP commands with PWM values from 280 to 320. The trick is to not dampen any shivering that does occur.

[nerdoug]

I swapped the cables for servo 0 and 1 at the PCA9685, and the symptoms stayed with the servo, not the controller port. I'm getting more convinced that it's the cantilevered weight applied to the hip servo's axis of rotation that causes the shivering. Haven't been able to come up with a solution, though.

[theAgingApprentice]

Perhaps we need to consider making the hip a shaft with a bearing in the body and a pulley or sprocket that can be driven by the motor free of lateral forces. I believe that such bears and pulleys exist for servos. Will see if I can find some links…

[theAgingApprentice]

Servo city has some options we could consider. Link: https://www.servocity.com/sprockets-chain/

[theAgingApprentice]

This page from ServoCity has more options and may be a better link: https://www.servocity.com/motion/. Alternatively we could use Servo blocks: https://www.servocity.com/servoblocks/ . Here is a video that shows them in action: https://www.youtube.com/watch?v=Jx6Wx7vPPv4 . It looks to me that the servoblock option is the most straight forward option. The actual kit from ServoCity: https://www.servocity.com/servoblock-standard-size-24-tooth-spline-hub-shaft/ . RobotShop only Carrie’s the 25T version which we cannot use, plus they are out of stock: https://ca.robotshop.com/products/large-hub-shaft-servoblock-25t-spline

[theAgingApprentice]

The ServoBlocks (https://www.servocity.com/servoblock-standard-size-24-tooth-spline-hub-shaft/) are a bit pricey at $29.99USD each but I think I can make most of the part for it. I think all we need to buy is:

1. 6 x 1/2" ID bearing flanges https://www.servocity.com/flanged-ball-bearings/?filter_bore=0.500%E2%80%B3+(1/2%E2%80%B3) which costs $3.99 for a pack of 2
2. 6 x Servo hub with 1/2" long shaft and 24T spline: https://www.servocity.com/24-tooth-servo-shaft-hub-0-500/ which costs $9.99USD each

If we decide to use servo blocks for the hip motors then the math works out like this for the above 2 options:

1. If we buy the full ServoBlocks from ServoCity the cost is 6 x $29.99 = $179.94USD per robot
2. If we buy the parts we need only from ServoCity the cost is 3 x $3.99 ($11.97) for bearings + 6 x $9.99 ($59.94) for a total of $71.91USD.

[nerdoug]

• Doug Elliott / VA3DAE / @.***

On Thu, Jan 12, 2023 at 4:15 PM The Aging Apprentice < @.***> wrote:

The ServoBlocks ( https://www.servocity.com/servoblock-standard-size-24-tooth-spline-hub-shaft/) are a bit pricey at $29.99USD each but I think I can make most of the part for it. I think all we need to buy is:

1. 6 x 1/2" ID bearing flanges https://www.servocity.com/flanged-ball-bearings/?filter_bore=0.500%E2%80%B3+(1/2%E2%80%B3) which costs $3.99 for a pack of 2
2. 6 x Servo hub with 1/2" long shaft and 24T spline: https://www.servocity.com/24-tooth-servo-shaft-hub-0-500/ which costs $9.99USD each

If we decide to use servo blocks for the hip motors then the math works out like this for the above 2 options:

1. If we buy the full ServoBlocks from ServoCity the cost is 6 x $29.99 = $179.94USD per robot
2. If we buy the parts we need only from ServoCity the cost is 3 x $3.99 ($11.97) for bearings + 6 x $9.99 ($59.94) for a total of $71.91USD.

— Reply to this email directly, view it on GitHub https://github.com/theAgingApprentice/HexaFloorRide/issues/73#issuecomment-1381002124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOZJYV4GK4UYCKNIL47UO3WSBYATANCNFSM6AAAAAATWGAW4I . You are receiving this because you were assigned.Message ID: @.***>

[nerdoug]

I could go either way, but maybe leaning towards the full servoblock, because I think it would be simpler and faster. I don't quite understand how we would mount it to the chassis, or what the implications are of lifting the legs by about an inch.

[theAgingApprentice]

12 full servo blocks on order. Shipping from ServoCity in the USA via FedEx international standard service. Will update this issue with tracking information when available. Will also create a new chassis CAD design and get measurements in hand for figuring out Inverse Kinematic math.

[nerdoug]

Let me know the total CDN$ and I'll give you some money.

• Doug Elliott / VA3DAE / @.***

On Sat, Jan 14, 2023 at 8:20 AM The Aging Apprentice < @.***> wrote:

12 full servo blocks on order. Shipping from ServoCity in the USA via FedEx international standard service. Will update this issue with tracking information when available.

— Reply to this email directly, view it on GitHub https://github.com/theAgingApprentice/HexaFloorRide/issues/73#issuecomment-1382736842, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOZJYS2UHZFP4245O2L4EDWSKRZNANCNFSM6AAAAAATWGAW4I . You are receiving this because you were assigned.Message ID: @.***>

[theAgingApprentice]

Order has shipped. Tracking information: 1ZAW55556899147523 (UPS (UPS® Standard))

Total cost on invoice = $397.01 USD. Total cost on Credit card = $546.98CDN. Not sure if additional brokerage fees will be charged by UPS, will know in the next few days.

[theAgingApprentice]

UPS estimates the shipment will arrive by the end of day on Friday. I am making some test pieces to ensure the new chassis design I have will work with the ServoBlocks. I believe I will need to use thicker aluminum stock to create the new thigh pieces as the mounting holes for #2-56 screws fit through 1/8" thick stock but M4 holes which the new mounting system uses requires thicker material. I'll likely use 1/4" stock.

[theAgingApprentice]

ServoBLocks arrived. There as a brokerage fee tat had to be paid. This brings the total to $546.98CDN + $151.93CDN = $698.91. Half of that is $349.45. The low CDN dollar plus shipping fees really hurt!

[theAgingApprentice]

Have the latest robot chassis with the new ServoBlocks installed. Did some initial testing and while I did manage t get 1 hip to shiver a little it was short lived (less than 2 seconds). If the issue is not completely solves, it is at least a lot better now. Will wait to see what Doug's experience is like.

[theAgingApprentice]

Doug is also seeing some shivering as well. He postulates that this only happens with servos that have no load on them. Doug suggests adding rubber bands to see if dampening the leg movements to see if that helps eliminate the remaining quivering we are seeing.

[nerdoug]

Interesting observation: If you use stp commands to put all legs in the vertical position, then there is no shivering of the hip servos, but the knees do shiver in much the same way. Futhermore, it only seems to happen to the legs on the left side of the robot. The right legs don't shiver on their own, and don't start to shiver if you nudge then, unlike the left side. Could this be a PCA9685 issue? Should scope the PWM line to one of the servos while it's shivering.

[theAgingApprentice]

I only saw shivering on one leg and it was not the hip. Cannot recall what side it was. Bottom line, I have seen similar behaviour so there must be some thing wrong.

Sent from Proton Mail for iOS

On Mon, Feb 13, 2023 at 12:16 AM, Doug Elliott VA3DAE @.***> wrote:

Interesting observation: If you use stp commands to put all legs in the vertical position, then there is no shivering of the hip servos, but the knees do shiver in much the same way. Futhermore, it only seems to happen to the legs on the left side of the robot. The right legs don't shiver on their own, and don't start to shiver if you nudge then, unlike the left side. Could this be a PCA9685 issue? Should scope the PWM line to one of the servos while it's shivering.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[theAgingApprentice]

1. Got new PCA9685 boards with an updated (beefier) transistor for reverse current protection
2. Replaced the 1000uF caps with 5600uF caps
3. Put a scope on ground and PWM signals and see the following:

• Noise on the ground line some of the time, other times the signal is quite clean.
• Shivering sometimes happens and sometimes does not.
• When putting load on the motor the power level on ground drops initially below 0.
• The more you load the motor the longer the below zero time is.
• The measuring technique used for these observations included using long ~24guage wire and a bread board for the scope which likely added resistance that affected the results we were looking at.

[theAgingApprentice]

1. Oscillator freq is set to 27Mhz
2. Servo Freq is set to 50 Hz (20ms per pulse)
3. Set up a new PCA9685 with all the skinny long wires for the Oscope gone
4. Rechecked signals and see the following:

• With no PWM to the motors the ground signal looks clean and there is no PWM to look at.
• With PWM activated, the hip is shivering
• The signal has a bit of an uptick after the pulse but is stable despite varying load.
• The +5 signal does dip down about .5 volts and the dip extends with less of a drop as more load is added.
• We can clean up the signal a lot but the shivering is the same.
• When we take the leg off the hip the shimmering stops immediately and cannot be restarted.
• So we can conclude the following:

1. Adding the servo block removed the axial loading but the shimmering remains.
2. Cleaning up the electrical signal with capacitors and higher gauge wire does not stop the shimmering.
3. Taking the extra cantilevered weight off the hip immediately stops the shimmering.
5. We therefore conclude that it is momentum (inertia) that is the root of the issue here The servos can hooked up cannot handle the weight and therefore overshoot the POT target and this causes endless hunting for the right POT position.

So we need a new physical design that eliminates this momentum issue. We can do this by

1. Reduce leg weight
2. Eliminate direct drive motor connection
3. Other?

We will proceed with developing this version of the robot and accept that shimmering is a consequence of the current design. In parallel we will start working non new leg designs for the next version of hex robot.