Teensy freezes when running 4 individual steppers

[baender]

Hi luni64 and everyone involved,

Short explanation for the context:

For a stage performance we are using a Teensy 3.6 together with TeensyStep to control 4 individual steppers. Target positions come in via Artnet (DMX via Ethernet) to control it with any theatre light controller. The stepper runs a linear slider that works as a small elevator for nails. In total we have more than 20 steppers, so one Teensy for each group of 4.

Approach

The code is based on the example RoseFunctionFollower. It uses a PID controller (only P for now) together with RotateControl::overrideSpeed() to allow on-the-fly speed changes, rather than set positions as they would be blocking.

For the minimal example below, the target position is a square wave with a period of 2 seconds (1 second max, 1 second min). Speed, acceleration and P were adjusted to get as close as possible to the target without overshooting. A debug Serial.print() allows us to monitor target and actual position (visualizing via SerialPlot application).

Problem

When running with all 4 steppers, Teensy freezes after some time. No more data are sent via UART. The time until everything freezes is not easy to describe. I took some measurements (manually), which you can find below:

5 min 05 sec
4 min 39 sec
2 min 33 sec
2 min 50 sec
1 min 06 sec
0 min 26 sec
0 min 11 sec
0 min 29 sec
0 min 09 sec
0 min 12 sec
0 min 36 sec
--- 10 min break ---
2 min 48 sec
1 min 12 sec
0 min 14 sec

When running several attempts back-to-back, the time seems to be decreasing. The freeze did not occur with only 1 stepper (at least for as long as we tested).

Setup

Plain Teensy 3.6 on a bread board. Nothing else.

Further steps

For any hint how to debug any further or even what the problem might be, I would be very happy to hear it.

Cheers, Daniel

#include <Arduino.h>
#include <SPI.h>
#include <array>
#include <TimerOne.h>
#include <TeensyStep.h>

constexpr uint32_t moveInterval    = 50000;      // in microseconds; interval at which rotation speed is updated (via TimerOne ISR)
constexpr uint32_t targetInterval  = 1000000;    // in microseconds; interval at which target position is updated (via elapsedMicros)

constexpr uint32_t maxSpeed        = 100000;     // steps per seccond
constexpr uint32_t maxAcceleration = 400000;     // steps per second^2
constexpr float pidProportional    = 7.3E-5;     // proportional factor for PID controller (no I or D used at the moment)

constexpr uint8_t fullDistance     = 168;        // in millimeters; length of linear distance
constexpr float mmPerStep          = 1.0/80.0;   // in millimeters per step; contains belt pitch and number of gear teeth

elapsedMicros targetTimer;                       // used for updating target position
float targetPosition               = 1.0;        // value between 0.0 and 1.0
volatile bool newTick              = false;      // ISR flag

// TimerOne ISR
void tick() {
  newTick = true;
}

class NailStepper {
public:
  NailStepper(uint8_t stepPin, uint8_t dirPin, int32_t motorMaxSpeed, int32_t motorMaxAcc, float pidProportional, bool printDebug)
  : m_stepper(stepPin, dirPin), m_pidProportional(pidProportional), m_printDebug(printDebug) { 
    m_stepper.setMaxSpeed(motorMaxSpeed);
    m_stepper.setAcceleration(motorMaxAcc);
    m_stepper.setPosition(0);                // set start position to 0
    m_controller.rotateAsync(m_stepper);     // start rotating
    m_controller.overrideSpeed(0);           // set rotation speed to 0
  }
  void setPosition(float newTarget) {
    m_targetPosition = static_cast<uint32_t>(newTarget * fullDistance / mmPerStep);
  }
  void move() {
    int32_t actualPosition = m_stepper.getPosition();
    int32_t error = m_targetPosition - actualPosition;
    float delta = m_pidProportional * error;
    float factor = std::max(-1.0f, std::min(1.0f, delta));
    m_controller.overrideSpeed(factor);
    if (m_printDebug) {
      Serial.printf("%d\t%d\t%.2f\n", m_targetPosition, actualPosition, delta);
    }
  }
private:
  Stepper m_stepper;
  RotateControl m_controller;

  uint32_t m_targetPosition;

  float m_pidProportional;
  bool m_printDebug;
};

NailStepper stepper1(6, 7, maxSpeed, maxAcceleration, pidProportional, true);
NailStepper stepper2(26, 25, maxSpeed, maxAcceleration, pidProportional, false);
NailStepper stepper3(16, 17, maxSpeed, maxAcceleration, pidProportional, false);
NailStepper stepper4(21, 22, maxSpeed, maxAcceleration, pidProportional, false);
std::array<NailStepper *, 4> steppers {&stepper1, &stepper2, &stepper3, &stepper4};

void setup() { 
  SPI.begin();
  Serial.begin(115200);
  while (!Serial);
  newTick = false;
  Timer1.initialize(moveInterval);
  Timer1.attachInterrupt(tick);
  targetTimer = 0;
}

void loop() {
  // Only update target position, no changes to rotation speed
  if (targetTimer >= targetInterval) {
    targetTimer = 0;
    for (const auto &stepper : steppers) {
      stepper->setPosition(targetPosition);
    }
    if (targetPosition == 0.0) {
      targetPosition = 1.0;
    } else if (targetPosition == 1.0) {
      targetPosition = 0.0;
    } else { // just in case
      targetPosition = 0.5;
    }
  }
  // Change rotation speed
  if (newTick) {
    newTick = false;
    for (const auto &stepper : steppers) {
      stepper->move();
    }
  }
}

[luni64]

Had a look at your code but didn't spot anything wrong. I tried it but didn't connect anything. I also removed the SPI include and begin since it is not used. It runs now for about 15min now without problem.

Maybe you have a hardware issue? Maybe some noise on the power line from the motors? Bad hub? Bad cable? Any high power supplies running close to the board?

[baender]

Hi luni, thanks a lot for the fast answer. I have nothing connected to the Teensy 3.6. Only a USB cable directly to the computer. No steppers, drivers or anything else. Could it be a compiler issue? I used the standard compiling settings from PlatformIO and VSC. The longest it took me once was 20min till the freeze.

[luni64]

It is still running here....

I can no imagine that this is a compiler issue. I really believe that this is related to some hardware issue. Anything getting hot on the board? Did you try other cables? hubs? Overclocked the board? Did you try to power the board from a stable supply via VIN?

If you think it is related to software, maybe you can generate a test app which fails more reliably? I can let the current one run here over night and report tomorrow...

[baender]

I did some more testing in the meanwhile.

First of all, I removed all Serial stuff (in software and USB cable). VIN of the Teensy 3.6 is now supplied by a Pololu 5V 3.2A step-down converter [1] which we use on our own PCB together with a 24V@100W power supply. The Teensy is all alone on a breadboard and the only load to the step-down converter. Also, it is a brand new Teensy 3.6.

Instead of Serial I used 3 digital pins to be able to indicate up to 8 states (6 were used). Two come at the beginning and end of loop(), another two at beginning and end of the first if-block, and another two at beginning and end of the second if-block.

Interestingly, the freeze does not occur when calling overrideSpeed() but later. Sometimes, the indicators show that the program stopped before looping (at the very end), sometimes it made it up to the beginning of the loop.

It is really hard for me to imagine that a power loss or brown-out occurs with the current power supply, even though I think it would suit the observation pretty well. Does the Teensy have an indicator whether there was a voltage drop on VIN? Do you have any other idea on what to test? Any idea is very much appretiated.

[1] https://www.pololu.com/product/3782

[baender]

Well, some more testing. The plot below shows what is being measured by an Adafruit INA219 attached to an Ardunio Nano for Serial communication. The INA219 is supplied by the Nano, so appart from shunt losses, no additional load is connected to the step down converter. In addition, I placed a 100uF low ESR capacitor right next to VIN and GND of the Teensy.

The plateau in voltage in the beginning is due to no-load (current is zero). The voltage line looks stable (peaks are LSB steps due to resolution) slightly below 5V, average current of around 66mA shouldn't pose a problem to either the step-down converter nor USB supply.

No pattern appears before the freeze at around -15sec.

*Edit: Measuring voltage with a Fluke 28 and no INA219 between Teensy and step-down (power is directly connected) I measure small fluctuations of < 10mV (between 5.038V and 5.047V).

[baender]

FINALLY it seems that I found some useful result!

After changing development environment from VSCode + PlatformIO to Arduino IDE, the above program (minus SPI and Serial.print()) is running without freeze for 45mins now, first try.

This would be a huge relief as I tried almost everything else that came to my mind, replacing the board with a brand new one, using different 5V sources, changing cables, trying on a Teensy 3.5, etc.

I had a look at the verbose compiling outputs from PlatformIO and Arduino IDE but I must admit, I do not know what differences to look for.

Assuming that this is the issue, I could either post the verbose compiler output here. Otherwise you might want to try out both environments to see if you can reproduce the freeze.

For now I am very happy with the solution of using Arduino IDE. However, it would be great if that could be solved also for PlatformIO, as it makes development much easier.

[luni64]

Sounds good! Easiest might be if you send me the hex file from your PIO build then I can try if it crashes here as well

This is really strange since I don't do anything special in the library. So, even if PIO is compiling with slightly different gcc settings it shouldn't matter much... Especially since I also don't use the lame Arduino IDE

shamelessAd _I'm using VisualTeensy which is also using vsCode but doesn't do anything behind your back. If you are on Windows I recommend to give it a try shamelessAd/

[baender]

This firmware is without Serial.print(). I am measuring voltage on Pin6 (Step for Stepper1). When everything runs as expected, voltage is changing rapidly (on my Fluke). As soon as the Teensy freezes (normally after 1-3 minutes), voltage is constantly 3.3V.

firmware.zip

Thanks for the hint with VisualTeensy. I am running everything on a Windows machine, however, this friend of mine for whom this project is built for has a Mac. For testing I will give it a try.

[luni64]

I can confirm that this firmware crashes after a short time.

[luni64]

I also complied with PIO, seems to run here as well (~15min)

platformio.ini

[env:teensy36]
platform = teensy
board = teensy36
framework = arduino
upload_command = c:\toolchain\TyTools\tycommander upload --autostart --wait --multi $SOURCE
build_flags = -DUSB_SERIAL_HID -DLAYOUT_US_ENGLISH

And here the compile log. Can you check if your installation uses the same packages?

Processing teensy36 (platform: teensy; board: teensy36; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/teensy/teensy36.html
PLATFORM: Teensy (4.13.1) > Teensy 3.6
HARDWARE: MK66FX1M0 180MHz, 256KB RAM, 1MB Flash  
DEBUG: Current (jlink) External (jlink)
PACKAGES:
 - framework-arduinoteensy 1.154.0 (1.54)
 - tool-teensy 1.154.210805 (1.54)
 - toolchain-gccarmnoneeabi 1.50401.190816 (5.4.1)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 93 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <SPI> 1.0       
|-- <TeensyStep> 2.1
|-- <TimerOne>
Building in release mode
Linking .pio\build\teensy36\firmware.elf
Checking size .pio\build\teensy36\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   2.9% (used 7508 bytes from 262144 bytes)
Flash: [          ]   3.4% (used 36120 bytes from 1048576 bytes)
Building .pio\build\teensy36\firmware.hex
Configuring upload protocol...
AVAILABLE: jlink, teensy-cli, teensy-gui
CURRENT: upload_protocol = teensy-gui
Uploading .pio\build\teensy36\firmware.hex
       upload@2504210-Teensy  Uploading to board '2504210-Teensy' (Teensy 3.6)
       upload@2504210-Teensy  Triggering board reboot
       upload@2504210-Teensy  Firmware: firmware.hex
       upload@2504210-Teensy  Flash usage: 36 kiB (3.4%)
       upload@2504210-Teensy  Uploading...
       upload@2504210-Teensy  Sending reset command

[baender]

Alright. So first, I built my project as is (no changes to platformio.ini). The compile log is identical to what you posted, except the RAM size (2.1% instead of 2.9%) and a little less Flash. The program froze as mentioned before. So I added build_flags = -DUSB_SERIAL_HID -DLAYOUT_US_ENGLISH. Then, it compiles with 2.9% RAM and indeed it was running without freeze (I let it run for only 10mins but it seems alright).

But here comes the thing. When removing the build flags and compiling, it still runs without freeze. I even created an entirely new project, switched to another Teensy 3.6 (one that was freezing with the old program) and built a project WITHOUT the build flags, and even this one is running for 20mins straight.

On the one hand it seems, that the problem is solved. On the other hand, I have absolutely no idea why a one time build_flag could achive such a thing. I'm thinking about uninstalling ArduinoIDE, PlatformIO and Teensyduino and reinstalling everything, just to see if I can reproduce the issue. Because as it seems now, I cannot any more since the one time I added the build_flags. I would let it run for at least 12h to really confirm that the issue is gone.

Compile log WITHOUT build flags:

Processing teensy36 (platform: teensy; board: teensy36; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/teensy/teensy36.html
PLATFORM: Teensy (4.13.1) > Teensy 3.6
HARDWARE: MK66FX1M0 180MHz, 256KB RAM, 1MB Flash
DEBUG: Current (jlink) External (jlink)
PACKAGES:
 - framework-arduinoteensy 1.154.0 (1.54)
 - tool-teensy 1.154.210805 (1.54)
 - toolchain-gccarmnoneeabi 1.50401.190816 (5.4.1)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 93 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <SPI> 1.0
|-- <TeensyStep> 2.1
|-- <TimerOne>
Building in release mode
Linking .pio\build\teensy36\firmware.elf
Checking size .pio\build\teensy36\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   2.1% (used 5632 bytes from 262144 bytes)
Flash: [          ]   3.4% (used 35464 bytes from 1048576 bytes)
Building .pio\build\teensy36\firmware.hex
Configuring upload protocol...
AVAILABLE: jlink, teensy-cli, teensy-gui
CURRENT: upload_protocol = teensy-gui
Uploading .pio\build\teensy36\firmware.hex

[baender]

Well, it seems that all this time I have been working with a slightly modified version of TeensyStep. I must have downloaded the correct version 2.1 via PlatformIO when changing platformio.ini to what worked for you and mistakenly assumed that this was the reason (starting to believe in Voodoo). My appologies for the circumstances. Closing this issue is probably for the best. Anyhow, thanks a lot for the support!