Teensy resets when using QF::poolInit without prior call of QS_INIT

[ChristianBauerEng]

Hello!

We've used qp-arduino for quite a while now on Teensy 4.0 and Teensy 4.1. As the project becomes more and more involved I wanted to update to the latest version to incorporate QSPY and (proper) UnitTesting.

Adapting the Teensy-example provided with this release was easy enough to get the expected Q-SPY output. But a few issues and questions remain:

• Why are QS::onCleanup(), QS::onGetTime(), QS::onFlush() and QS::onReset() needed, even when Q-Spy is disabled (QS_ON not defined). This seems to be inconsistent with QS::onStartup() and QS::onCommand(), which do not necessarily have to be implemented. This is not a problem, just a question, though.

• When QF::poolInit(...) is called before QS::onStartup() runs, Teensy resets itself. So far I've tracked it down to these BSP-calls in QS::onStartup(void const * arg):

  static uint8_t qsTxBuf[1024]; // buffer for QS transmit channel (QS-TX)
  static uint8_t qsRxBuf[128];  // buffer for QS receive channel (QS-RX)
  initBuf  (qsTxBuf, sizeof(qsTxBuf));
  rxInitBuf(qsRxBuf, sizeof(qsRxBuf));

  If initBuf and rxInitBuf are not called before initializing the event pool (QF::poolInit(...)) the MCU resets. This means, that without QS_ON no event pool can currently be used, as this surpresses the call to QS::onStartup().

To reproduce, add these lines to the Teensy setup() method after BSP::init();:

DMAMEM static QF_MPOOL_EL(uint64_t) BigPoolSto[50];
QF::poolInit(BigPoolSto, sizeof(BigPoolSto), sizeof(BigPoolSto[0]));

As long as QS_ON is defined, the code should run fine. However, if QS_ON is not defined, the MCU should crash and reset itself.

Here is the code I used to test and reproduce the problem. If you extract it into /libraries/qpcpp_arm-cm/examples/blinky-qspy-teensy4/ it should be directly executable with PlatformIO. teensy.zip

[quantum-leaps]

Hi Christian, The basic answer to all your questions is that the QP/C++ adaptation for Arduino is suboptimal, especially the QS software tracing support. The adaptation has to comply with the Arduino build process, which supports only one "build configuration" and has many other limitations.

This basic constraint (compliance with Arduino build process) has the following consequences:

Why are QS::onCleanup(), QS::onGetTime(), QS::onFlush() and QS::onReset() needed.

The Arduino build process pulls in all the files in the "library" directory, including the QS files. This code makes calls to the "QS callbacks" and therefore they are pulled in as a consequence. Outside of Arduino, the "normal" build process for QP/C++ uses different "build configurations", so only the "Spy build configuration" includes the software tracing instrumentation.

When QF::poolInit(...) is called before QS::onStartup() runs, Teensy resets itself...

This is most likely because QF::poolInit() tries to produce QS trace records, while QS is not yet initialized.

This means, that without QS_ON no event pool can currently be used,

This is a dependency (bug) introduced by the hasty recent "integration" of QS software tracing into the QP/C++ Arduino. Please note that "QS_ON" is actually an "Arduino-special". The real macro to control tracing is Q_SPY, but for the compatibility with the Arduino build process it could not be used, so "QS_ON" has been introduced on top. Also, the actual QS output with the Serial.print() interface is very inefficient and causes completely wasteful copying of data from the QS TX buffer to the Arduino Serial buffer.

At this point, it's clear that all of this should be redone. For real work, however, I would not use the "Arduino crutch" but rather a direct build, as supported in the "real" QP/C++. This would also be useful for unit testing, which should run from the command line and not through Arduino IDE. (Also, incremental builds from the command-line would be 10x faster than the Arduino builds, which re-build stuff completely unnecessarily). I understand that the problem is the Arduino library for accessing the hardware, which would have to be somehow adapted.

I hope that my comments make sense to you.

--MMS

[ChristianBauerEng]

Hi Miro,

thank you very much for your clear answer!

Your response makes perfectly sense and I understand the limitations. We've abandoned the Arduine IDE long ago and are now using PlatformIO, which alleviates many of the problems.

I tried to look into it, but replacing Q_SPYwith QS_ONin this region alone does not solve the issue. https://github.com/QuantumLeaps/qp-arduino/blob/cf386765434e96f4b7cd230ee2ed868d3aca6990/libraries/qpcpp_arm-cm/src/qf_dyn.cpp#L104-L111 It is probably a combination of several locations where these guards would have to be changed. But that doesn't seem like an unsolvable road block so for now I'm fine with it.

The "Arduino crutch" as you've put it so nicely has also been bothering me for quite a while now, but the number of libraries we use and depend on it is simply too large to easily abandon it.

Since we're still well away from any prduction-type of release I'll just keep QS activated for now.

Kind regards,

Christian

[ChristianBauerEng]

Sorry for re-opening this issue for a quick question: Are there any plans to support QUtest on the Arduino platform as well?

I've created a custom test-runner for PlatformIO which compiles and uploads a test fixture, opens qspy and forwards any data received over the programming port to qspy. So far it works like a charm and helps a lot!

Running and connecting qutest to the qspy-instance works as well, but of course fails to run actual tests. Would be nice to have the entire toolchain in place for automated unit testing. Do you think qutest will be available in the future anytime soon?

Kind regards,

Christian

[quantum-leaps]

The main objective of the QP adaptation for Arduino is the same as Arduino itself: to provide a prototyping platform. For example, Arduino does not even provide any hardware debugger interface (e.g., JTAG) and the primary way of troubleshooting Arduino "sketches" is through "debugging by print".

Are there any plans to support QUtest on the Arduino platform as well?

In view of these basic design objectives, there are no plans to extend the QP support on Arduino with QUTest or other such features. How would that even work? I mean, true TDD (Test Driven Development) requires automation and the ability to execute the tests from the command line. AFAIK, Arduino does not support command-line builds...

I hope that these comments make sense to you...

--MMS

[ChristianBauerEng]

Thank you again.

Your answer makes perfectly sense if you think about the Arduino IDE. There it would indeed not make sense to incorporate Qutest.

However, alternatives like PlatformIO or VisualMicro (which still use the Arduino framework) offer the full set of development tools (hardware debugger, automated unit testing frameworks, etc) while still making use of the huge Arduino library ecosystem.

But from your answers I understand that the Arduino-specific port of QP was only necessary due to the way the Arduino IDE works. If that turns out to be correct we'll try to switch to the "proper" qpcpp with the full set of features.

[ChristianBauerEng]

I was indeed able to get the "proper" qpcpp running on the Teensy through PlatformIO. I have yet to test all the functionality, but at least Blinky is running fine. Once it's cleaned up a bit I will provide a repo-fork (of the modified qpcpp repo) and close this issue. The fork will contain the PIO configuration for a Teensy running on the Arduino platform and a custom test runner for automated unit testing. I feel a bit stupid for not trying this right away, but maybe someone will stumble over it in the future and find it helpful.

[quantum-leaps]

Hi Christian, I'm glad to hear that you made some progress with running QP/C++ (the "real" thing) on your Teensy board.

When you're ready to release your examples, you might want to advertise them in the "contributed" repository.

--MMS

[ChristianBauerEng]

Even though my current work has now hardly anything to do with the Arduino port any more, I hope it's fine to continue the discussion here...

I've attempted to create a qutest-port for the Teensy by going through some of the code used for other targets. It can currently be found here: qutest_port.cpp The corresponding test-fixture for the blinky-test is here. The test script and qutest.py were copied directly from the example-directory.

However, when I run the unit test (first QSPY, then QUtest) qutest fails to reset the target. As I understand it QUTest sends the reset command to QSpy, but QSpy never responds (QSpy._receive() times out), which happens around here:

      else: # running an embedded target
          QUTest._have_target = True
          QUTest._have_info = False
          if not QUTest._is_debug:
              QSpy._sendTo(struct.pack("<B", QSpy._TRGT_RESET))

      # ignore all input until have-target-info or timeout
      while QSpy._receive():
          if QUTest._have_info:
              break

This produces QSpy-output similar to this:

What is strange is, that sometimes QSpy shows the TstPause entry, sometimes it doesn't. And even in the very few instances where QSpy responded to QUtest with the target_info, running continue_test and setting filters did not seem to have an effect and did not cause QSpy to produce more output.

This all seems to point towards an error in the qutest_port. I would be really happy if you could give me a small push in the right direction or at least some idea on where to look.

Cheers!

[quantum-leaps]

Hi Chris, Your test stops at the QS_TEST_PAUSE() macro. Please read about "Pausing the Test" and more generally, about the Run-to-Completion Processing inside QUTest tests. --MMS

[ChristianBauerEng]

Hi Miro,

the problem was, that I had not implemented the RX-channel in the qutest-port (facepalm). I had adapted one of the existing example-ports, which didn't have the RX implemented either. So the MCU simply never got the reset-command from qspy. That's fixed now. Without QUtest it used to work, because the RX is part of the onIdle() function of QV. So it took me longer to find the missing RX functionality than I'd like to admit...

Anyway, since the Teensy is USB-based, resetting the MCU meant, that the COM-connection is lost which caused QSPY to terminate and thus QUtest to fail. I've worked around that by implementing a buffer-layer between the MCU and QSPY, which forwards bytes from one entity to the other and automatically re-connects the COM port, as soon as it is available again. Since that can take a couple of seconds, though, I had to increase the timeout in qutest.py, which may have some side-effects that I'm probably not aware yet.

The resulting custom test runner became a lot more complicated than I had anticipated. It's not pretty and not very stable, but at least for now it seems to be more or less working. Maybe this will help someone in the future to not make the same stupid mistakes as me :-)

Regards,

Christian