Missing steps ESP32 S2 in high dynamic usage

[ChrGri]

Hello and thank you for this awesome library.

This library is used with an ESP32 S2 Mini in my DIY active pedal (https://github.com/ChrGri/DIY-Sim-Racing-Active-Pedal/blob/main/Arduino/Esp32_S2/Main/Main.ino). Here, a Loadcell measures force applied to the pedal and uses a closed loop stepper (https://www.omc-stepperonline.com/de/nema-23-integrierter-easy-servo-motor-180w-3000rpm-0-6nm-84-98oz-in-20-50vdc-buerstenloser-dc-servomotor-isv57t-180) to adjust the pedal position. Unfortunately, after some usage, the pedal seems to lose its neutral position, thus the pedal tilts more and more to the ground and I'm trying to understand the root cause of this problem.

The cycle time of the code is approx. 130 us and depending on the loadcells reading, a new non-blocking moveTo(..., False) is executed. The position and direction might significantly differ from the cycle before. Stepper speed and acceleration are configured rather high as required by the application.

Step+ and Dir+ of the motor are wired to 3V3 and Step- and Dir- are wired to the GPIOs. I wonder, if applied in this configuration, some stepper code logic needs to be changed from rising to falling edge? Other remarks of potential solutions are highly welcome.

BR

[gin66]

If I understand the datasheet correctly, then the DIR- and PUL- should be connected to GND.

In order to support the debugging, I would advise:

1. which position does FastAccelStepper report ?
2. if you remove the load, do you still observe the deviation after some time ?

[ChrGri]

If I understand the datasheet correctly, then the DIR- and PUL- should be connected to GND.

In order to support the debugging, I would advise:

1. which position does FastAccelStepper report ?
2. if you remove the load, do you still observe the deviation after some time ?

Thank you @gin66 for the fast reply. I've tried grounding the DIR- and Step- pins first, but then the ESP32 was not starting. Therefore I used the GPIO as ground.

Calling "stepper->getCurrentPosition();" gives the drifted position (e.g. 0 although the pedal clearly is out of the initial position) unfortunately. I will try without load next.

BR

[ChrGri]

@gin66 I've removed the load from the stepper. When I ask for high velocity & acceleration movements with direction changes in between the cycles, sometimes the stepper moves in the wrong direction. The behaviour gets worse, when decreasing the maximum acceleration, e.g. from stepper->setAcceleration(1e7) to stepper->setAcceleration(1e4) makes it worse.

The observation can be described as follows: 1) ask for a movement of DeltaX 2) stepper moves to DeltaX 3) ask for a movement of -DeltaX 4) stepper moves to 2*DeltaX, although movement to 0 was requested.

I've build a new µC where the Dir- and Step- are connected to GND and Dir+ and Step+ connected to the GPIOs as suggested --> no difference.

[gin66]

The ESP32 GPIO output is 3.3V, but your motor DIR+ and STEP+ need >=5V. Perhaps this could be a reason.

BTW: do you have an oscilloscope, where you can measure the DIR/STEP lines ?

Looking at your acceleration values: do you need acceleration at all?

[ChrGri]

I've read a article from stepperonline which mentions that they have tested 3V3 for their closed loop system (https://www.omc-stepperonline.com/support/do-your-stepper-drivers-accept-33v-signal-voltage-for-arduino-and-raspberry-pi).

I do have a logic analyzer, which I'm using next.

I don't need acceleration. How do I disable it in the library?

---

EDIT: Logic analyzer says, that DIR pin output sometimes keeps beeing LOW although, I'm giving a command in which it should change to HIGH.

[gin66]

do you have a short working code, which reveals the issue, so I can test it here ? Without all the peripherals in the code, which I do not have. This means, having only the stepper code would be of great help to see, if there is any problem with the lib.

Depending on the logic analyzer, perhaps the load of the driver is just too high and even the logic analyzer detects sometimes low. Anyway, the manufacturer does not recommend 3.3V operation. Already some ground shifts or long wires may impact the outcome.

I have thought for some time to disable acceleration altogether e.g. by setting acceleration to 0, but haven’t implemented it.

[ChrGri]

I've tested without the stepper driver cables attached and observed similar behaviour.

I will prepare some test code without the other dependencies now.

---

EDIT: Testfile has been uploaded here

---

EDIT 2: Here is a screenshot of the logic analyzer output vs the moveTo(x) command plot. Logic analyzer channel 0: DIR channel Logic analyzer channel 1: STEP channel

The logic analyzer plot does not change when stepper motor is connected ot not.

The command plot shows the position to which the stepper should move. Instead of following the sawtooth pattern, the stepper continues spinning in one direction. From the logic analyzer channel 0 output one can see, that no direction change to the stepper is requested, although it is requested by the moveTo() call.

[gin66]

Thanks for the measurement and code example. This is very helpful

I have modified the code with some comments: (https://github.com/gin66/FastAccelStepper/blob/20a8301b4b8c4ee2bbb4e657049f4c0df26045b7/examples/Issue174/Issue174.ino)

The issue is, that you run the loop with cycle time of couple of 100 us. The stepper gets every cycle a new position. Most of the positions, the stepper task does not even see. The stepper task looks every 4ms for commands and all former position requests are overwritten. That's why, I have added print out of the stepper->targetPos(). This way you can see, what the stepper is intending to run to.

The max speed in your system is 13333 steps/s. So at full speed at 130us loop cycle time, the stepper can run only 1 or 2 steps. But every loop requests to run 500 steps during that time. This does not work. Consequently: the stepper needs time to run.

[ChrGri]

Thanks for the testcase. Hardware wise, the stepper can handle up to 4k RPM and 300kHz step rates.

Given that the steppers trajectory is updated every 4ms and the sawtooth from the test has a period of 2.6ms = 130us/cycle * 10000 steps/tooth /500steps/cycle, I would have expected the stepper to be mostly stationary and move forward and backwards only slightly. But since the stepper kept rotating in one direction, could it be, that the ramp generator planner loses track of its own states somewhere?

Real world usage should not be as demanding as this testcase. However, high frame rates are required, otherwise the pedal will start to oscillate.

[gin66]

There is still the possibility, that on esp32 something goes wrong or something is not ok with the interface to that motor, which causes position divergence. Today I have made the test using simavr, which emulates an AtMega328p, and it has run well. Testing on esp32 takes more time as I need to prepare and attach the esp32 hardware.

On ESP32 the rmt module is used, which I believe to have better performance than mcpwm/pcnt, but rmt is most likely less in use in the field. In regard to steps lost: this should not happen at all, if neither ForceStopAndNewPosition nor setCurrentPosition has been called. But I still have doubts about the motor interface 3,3V/5V, so I would rather start with this.

BTW: cool project of yours and nice video.

[gin66]

Pin losses could be caused by DIR/STEP timing. FastAccelStepper supports to add a delay on direction change:

setDirectionPin(dirPinStepper, false , MIN_DIR_DELAY_US);

Perhaps this helps

[ChrGri]

Thank you very much. I will try the test on an Arduino Nano later that day.

Do you have an idea which delay value on direction changes I should try with the ESP32?

Thank you for this library and if you need an ESP32 S2 Mini (15€/3 units) for testing, I'd be happy to help e.g. via Amazon wishlist or PayPal.

---

EDIT: Repeated the updated test with an Arduino Nano and had similar results as with the ESP32. Here is the Logic Analyzer output:

Channel 3 is for debug purposes and is high, when the target position wraps back to 0, see logic:

Channel 4 is the Dir+ signal. Channel 5 is the Step+ signal.

Expected behaviour: Channel 4 should follow channel 3 somehow. Observed behaviour: Channel 4 is always LOW.

Played with cycle time and acceleration. If acceleration and cycle time are high, direction changes are applied as desired. If cycle time and acceleration are reduced, at some point the direction changes aren't sheduled and the motor will keep rotating in one direction. So, there might be a bug in the common part of the software for ESP32 and Atmega.

[gin66]

In your example code, the acceleration is 100. This means direction changes will not happen fast and your assumption "Channel 4 should follow channel 3 somehow" is not valid.

I have updated the example in the Issue174-branch to match it to your code.

I think the scaling in the image is not correct, but at least it is clearly visible, that DIR A is changing:

[ChrGri]

Thank you for the quick reply. The Dir changes are only applied on my controller, if the acceleration and cycle times are high. If cycle time is e.g. 130us and acceleration is low, the motor keeps spinning in one direction and is drifting away. I would have expected that the motor will mostly stay in position due to limited acceleration. Please repeat the test with cycle times of 130us and check your outcome.

Btw. using the serial print commands might modify the test outcome as it typically costs some ms itself.

[gin66]

The code, which I have used is here.

Could you please check and tell me, which values I should change ?

[ChrGri]

@gin66 I'm so sorry. I've to withdraw my statement, that it's not working on the Atmel as well. Uploaded the same C code to Arduino Nano and ESP32 S2 mini and let them run in parallel. The output of the logic analyzer is different for the Arduino and indeed shows Dir changes. For the ESP32 this is not the case.

See Ch0 vs Ch5.

Arduino Nano: Ch 0: DIR+ Ch 1: Debug Ch 2: STEP+

ESP32 S2 mini: Ch 5: DIR+ Ch 6: Debug Ch 7: STEP+

[gin66]

OK. So I need to test directly on HW and the atmel simulator is not of help anymore.

Could you please do me a favor and check within the loop for stepper->rampState() ? I like to check, if the issue is related to the ramp generator or the low level driver. That ramp state value may of help.

[ChrGri]

Sure. I've added the requested command like seen below:

Here are the two log files: AruinoNano.log Esp32_S2_mini.log

Not sure whether this is important or not, but the Dir pulses differe in length a lot. Ch 2 is Arduino and Ch 7 is the ESP:

I've setup the cycle time to be 500us for both µC in that example.

[ChrGri]

And just for completeness, here is the logic analyzer output for cycle time of 500ms and acceleration=1e8 steps/s^2

The direction change can bee seen for ESP (CH 5) and Arduino (CH 0).

[gin66]

Thanks for quick action.

The ramp states are defined like this:

// The defined ramp states are:
#define RAMP_STATE_IDLE 0
#define RAMP_STATE_COAST 1
#define RAMP_STATE_ACCELERATE 2
#define RAMP_STATE_DECELERATE_TO_STOP 4
#define RAMP_STATE_DECELERATE (4 + 8)
#define RAMP_STATE_REVERSE (4 + 16)
#define RAMP_STATE_ACCELERATING_FLAG 2
#define RAMP_STATE_DECELERATING_FLAG 4

// And the two directions of a move
#define RAMP_DIRECTION_COUNT_UP 32
#define RAMP_DIRECTION_COUNT_DOWN 64

So there are the following states in the esp32 log:

34 count down+accelerating
44 count down+decelerating
52 count down+reverse
66 count up+accelerating
76 count up+decelerating
84 count up+reverse

This means:

1. The stepper should go in both directions
2. At no point, the stepper reaches max. speed

And from your latest info, I understand, the stepper is changing direction, too.

So I currently cannot find any issue.

Could you please change the code, that the ramp is not executed so superfast, which gives FastAccelStepper a position to follow ? Please try instead this code, which provides the stepper with the positions to run the ramp 10 times per second:

PositionNext = (currentTime / 10) % 10000;

[ChrGri]

And from your latest info, I understand, the stepper is changing direction, too.

For the ESP: Only for the example where the cycle time was increased to 500ms and the acceleration increased to 1e8. The post before -the one with the attached log files- was created with cycle times of 500us. Stepper wasn't changing direction there.

For the Arduino Nano: Stepper is changing directions for both cases.

[gin66]

OK. Now I understand. Thanks for clarification.

Hope later today I can check on the esp32 (without S2)

[ChrGri]

Good point, I've done the same test with the esp32 (without S2) and it's running great. Direction changes as expected.

Arduino Nano: Ch 0: DIR+ Ch 1: Debug Ch 2: STEP+

ESP32: Ch 5: DIR+ Ch 6: Debug Ch 7: STEP+

Thus, only the esp32 S2 seems to be affected by the issue.

[gin66]

The S2 uses RMT module and the stock esp32 a mcpwm/pcnt combo. But there is a possibility to force the use of the rmt module even for stock:

#define DRIVER_MCPWM_PCNT 0
#define DRIVER_RMT 1
#define DRIVER_DONT_CARE 2
  FastAccelStepper* stepperConnectToPin(uint8_t step_pin, uint8_t driver_type);

Just add the parameter DRIVER_RMT to the stepperConnectToPin(). On the esp32s2 will give a compile error.

[ChrGri]

Good to know. I've flashed the esp32 (without S2) with DRIVER_MCPWM_PCNT and DRIVER_RMT. CH5 is Dir+, CH6 is Step+ and CH7 is the debug signal. CH0-2 are the Arduino Nano signals for reference.

stepper = engine.stepperConnectToPin(stepPinStepper, DRIVER_MCPWM_PCNT); gives this result:

stepper = engine.stepperConnectToPin(stepPinStepper, DRIVER_RMT);gives this result:

Link to the testscript with selected parameters.

The observation is, that RMT does not generate the Dir change, whereas the MCPWN generates Dir changes as requested.

[gin66]

With esp32-hw and your app, I could reproduce the issue with the rmt-module.

Funnily, there was even a comment in the rmt-module about the dir-change and stating an assumption. Apparently that assumption has not been met in your application. I have now applied a quick-fix, which allows the motor to change direction. I am just not sure, if this causes steps lost. That's why I have added pulse counter support for esp32s2, which can support the detection of lost pulses.

[gin66]

the esp32 based hw tests indicate, that steps are lost with the rmt-module. It will take more time to fix this problem.

[ChrGri]

You are absolutely awesome. I've flashed the update and it's already miles better than before.

When flashing my testscript to my pedal, I indeed can confirm the lost steps. With every sawtooth the steppers 0 position moves constantly further from it's initial position. Wanted to count the forward and backwards steps via logic analyzer next.

[ChrGri]

For debug purposes, I've exported the data from my logic analyzer to a CSV file and decoded the data via python.

Perhaps this is helpful for you as well.

---

EDIT: Here is a zoomed in image of the stepper position of the sawtooths for the sequential moveTo(0) command:

MCPWM: Observation: local minima of the stepper positions are all of equal value.

RMT: Observation: local minima of the stepper positions are out of alignment.

I've checked the timing of the Step+ and Dir+ signal because I though, maybe the Dir+ signal is jittered and therefore some steps are triggered in the wrong direction, but thats not the case --> The number of step pulses must be wrong.

Therefore, I've looked more closely into the steps burst -these 500 step increments from the test file- and found, that MCPWM produces almost equally spaced steps and RMT is missing some steps at the end of the burst.

500 step burst on MCPWM:

500 step burst on RMT:

[gin66]

Thanks for the detailed investigation. The mcpwm/pcnt combo runs pretty well, while the rmt-version has some issues. In the meantime I have reworked the code and believe, that changes in direction is now better managed and step pauses as in your pictures are avoided.

Most of my hw tests are passing. Just two similar test cases are constantly failing.

So step loss has improved, so I have decided to upload the current code. Hope you can confirm an improvement, too.

[ChrGri]

Thank you very much. I've flashed with the update library. It's much better, yet not perfect.

RMT with 500ms delay in between the bursts:

Observation: No steps lost

RMT with 50ms delay in between the bursts:

Observation: No steps lost

RMT with 5ms delay in between the bursts:

Observation: Steps are beeing lost, position is drifting.

MCPWM with 5ms delay in between the bursts:

Observation: Steps are beeing lost, position is not drifting.

For the tests, the stepper limits have been parameterized as follows: stepper->setSpeedInHz(33333) stepper->setAcceleration(1e8).

The required steprates to follow the burst with the given delay of [500ms, 50ms, 50ms] have to be [1k, 10k, 100k]steps/s. Ignoring acceleration here.

I feel like, that RMT behaves perfect when the requested step rates are lower than the parameterized limit. But steps are lost and the position drifts when the requested steprate is higher than the parameterized limit. MCPWM, seems not to drift but oscilation around the center. However, the requested positions are in the range of [0,9500]. The plots show, that only 3000 steps per sawtooth are executed. I still need to verify whether the position will drift outside that [0,9500] position window. The patterns for MCPWM and RMT might be valid otherwise.

Does this observation relate to the failed testcases?

[gin66]

Thanks for the measurements and it fits for the rmt. The two failing test cases are faster versions of a test, which is passing. So I need to focus on that.

For mcpwm/pcnt I have not observed step loss for a while and with my test cases. After rmt is ok, I will check that again. The issue with mcpwm/pcnt is, that high speed directly followed by a pause may lead to step loss under higher cpu load, because the interrupt has to stop the pwm right before the next step. At high speed of 100ksteps/s, this is <<10us. Using deceleration with not so high values, any possible overruns due to delayed interrupt service at high speed can be recovered, because the pulse counter detects this.

[ChrGri]

@gin66 I'm using the ADC lib to gather ADS1256 readings via SPI. Calling stepper = engine.stepperConnectToPin(stepPinStepper); corrupts the data reading or breaks the SPI communication to the ADC. Don't know exactly. The old implementation did not have that problem.

Apparently the call order has an influence. It does not work, when I initialize the ADC first and the stepper afterwards. It works, when I initialize the stepper first and the ADC afterwards.

It's not a blocking point for my project, since I only need to change the call order to make things work. But, perhaps, something you may want to keep an eye on.

[gin66]

is it using Pin 18 ? I have forgotten to disable the probe for toggling this port pin in the rmt driver:

#define TEST_PROBE_1 18

[ChrGri]

Oh, yes it is. Apart from that, still steps lost in high dynamic real world use, as your failed testcases may have already reported.

[gin66]

The new version passes all tests.

As I had once a test failure, perhaps there is still a sporadic issue, which needs to be monitored further. Please try the latest version

[ChrGri]

Thank you. Flashed the the code to the pedal. Unfortunately, it seems to perform a lot worse than it did before. From observation I would say, that the direction signal isn't following the requested direction.

I'm gonna try to write a testcase which is as close as possible to the application.

---

Edit: I've added another testcase, where the stepper oscillates with increasing frequecy. At some point, the stepper starts to move in the wrong directions.

[gin66]

Do you observe any step loss aka position drifting ?

[ChrGri]

I think that it's fine, as long as the requested step rate is lower than the parameterized maximum steprate. Once the requested steprate gets to large, it's much more drifting than before. I reperformed the sawtooth testpattern with [300, 5]ms delay between the bursts.

Esp32 S2 Delay 300ms

Observation: Position follows the sawtooth pattern

Esp32 S2 Delay 5ms

Observation: Position drifts away, significantly worse than before

For comparison, this was RMT with 5ms delay from yesterday:

---

Edit: One thing which is strange though, I've reduced the stepper->setAcceleration(1e9) -->stepper->setAcceleration(1e6) and the pedal works perfectly fine now. The 5ms sawtooth testcase however fails.

[gin66]

I have added your example with modification as Issue174.ino under examples. Here you can see, how to attach a pulse counter to check, if steps are lost.

The pulse generation runs flawless with acceleration 1e6.

With 1e9 there are immediately deviations and the esp32 stalls even after a short while, which is even more weird. The log out of reset is:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12680
load:0x40080400,len:2908
entry 0x400805c4
Setup stepper!
Return to 0 from 491
=> FAIL with pcnt=
2
Return to 0 from 439
=> FAIL with pcnt=
4
Return to 0 from 359
=> OK
Return to 0 from 243
=> FAIL with pcnt=
20
Return to 0 from 109
=> OK
Return to 0 from -37
=> FAIL with pcnt=
-5
Return to 0 from -170
=> FAIL with pcnt=
-1
Return to 0 from -305
=> OK
Return to 0 from -423
=> OK
Return to 0 from -486
=> OK

[ChrGri]

That's weird indeed. Perhaps a numerical issue? Are the failed testcases RMT exclusive?

[gin66]

still have an issue in the rmt-driver. This will take some more time to fix

[ChrGri]

still have an issue in the rmt-driver. This will take some more time to fix

You already did an awesome job. Let me know, if you need any help.

[ChrGri]

FYI: I've had significant amount of step loses on the pedal, when calling (a) for a move to the stepper->getCurrentPosition() or (b) same target position over and over again (150us cycle time). Don't know yet if (a) or (b) created that problem. Problems are gone with this command: https://github.com/ChrGri/DIY-Sim-Racing-Active-Pedal/blob/33c2687d5ded583c01289e7f45100538fee369e3/Arduino/Esp32_S2/Main/Main.ino#L544

Haven't tried the command on the testcase yet.

[gin66]

Good that you have found a solution.

Anyway, I have set up my esp32 with logic analyzer and modified your test case. Several bugs identified and fixed. Hope it applies to esp32s2, too.

So I have released 0.30.1 as I believe it to be much better. All esp32 hw based tests are passed and your test case (modified and stored as examples/Issue174/Issue174.ino) do not show any pulse loss anymore.

[ChrGri]

Thanks a lot. I've tried back and forth. With the current mainline head, for me, the stepper isn't moving anymore.

[gin66]

You are using the esp32 or esp32s2 ? I can only test the esp32 and the examples/Issue174/Issue174.ino runs well. No steps lost anymore.

[ChrGri]

I've only tried on my esp32 s2 today. I will try on a regular esp32 tomorrow.

[gin66]

Please try to apply this patch:

diff --git a/src/StepperISR_esp32_rmt.cpp b/src/StepperISR_esp32_rmt.cpp
index d459069..60899ba 100644
--- a/src/StepperISR_esp32_rmt.cpp
+++ b/src/StepperISR_esp32_rmt.cpp
@@ -492,6 +492,8 @@ void StepperQueue::startQueue_rmt() {

   // This starts the rmt module
   RMT.conf_ch[channel].conf1.tx_conti_mode = 1;
+
+  RMT.conf_ch[channel].conf1.tx_start = 1;
 }
 void StepperQueue::forceStop_rmt() {
   stop_rmt(true);

Perhaps the ESP32S2 needs the tx_start, which the esp32 does not need.

[ChrGri]

Happy to confirm that patch is working :)

In a short test, the pedal felt great. I will spend more time this week trying to provoke step losses.