Most important LEGO import does not work

[maarten-pennings]

The LEGO IDE imports MSHub. Why does that not work on uPIDE?

# file demo.py
from mindstorms import MSHub

I get in the console (why in sensors.py?)

sensors.py, line 1, in <module>
RuntimeError: maximum recursion depth exceeded

[harbaum]

I'll have a look at that. Does this happen in REPL or only if you create and run a file? If it happens in REPL, does this also happen in REPL when not using µPIDE but e.g. Hyperterm instead?

[maarten-pennings]

Good suggestion. This is in the REPL in uPIDE

>>> from mindstorms import MSHub
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mindstorms/__init__.py", line 1, in <module>
  File "_api/__init__.py", line 1, in <module>
  File "_api/lightmatrix.py", line 1, in <module>
  File "system/__init__.py", line 1, in <module>
  File "system/callbacks/__init__.py", line 1, in <module>
  File "util/error_handler.py", line 1, in <module>
  File "protocol/__init__.py", line 1, in <module>
  File "protocol/rpc_protocol.py", line 1, in <module>
  File "util/sensors.py", line 1, in <module>
RuntimeError: maximum recursion depth exceeded
>>> 

So, uPIDE.REPL also gives error.

When I use RealTerm, after a reset, I first send ^C, then the import, this works fine

{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -13, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
{"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [
Traceback (most recent call last):, "", 0]}                                     
  File "main.py", line 8, in <module>                                           
  File "hub_runtime.py", line 1, in start                                       
  File "event_loop/event_loop.py", line 1, in run_forever                       
  File "event_loop/event_loop.py", line 1, in step                              
KeyboardInterrupt:                                                              
MicroPython v1.14-893-gbae6ff2ee on 2021-11-04; LEGO Technic Large Hub with STM3
2F413xx                                                                         
Type "help()" for more information.                                             
>>> from mindstorms import MSHub                                                
from mindstorms import MSHub                                                    
>>>     

[harbaum]

Can you give a complete example that needs this?

[harbaum]

If you press ctrl-d in repl then the import works afterwards.

This is actually tricky to debug as the LEGO firmware itself has has been running and also µPIDE has interacted with the brick. So somehow the LEGO brick won't do this import again and ends up in some recursion somewhere. We cannot see where since LEGO does not publish the source code.

So I'd like to see a real-life example that needs this. So I understand what's lost if that import does not work. When and what for is this import needed?

[maarten-pennings]

I was very impressed with how smooth uPIDE handles the LEGO hub. Running a program without storing it in a file, knowing when it terminated, uploading files. I don't understand yet how you do that, nor do I understand what kind of firmware the hub runs at startup, generating those funny lines {"m":0,"p":[[75, [0, 0, -14, 0]], [0, []], [0, []], [0, []], [0, []], [0, []], [.

But, your question "So I'd like to see a real-life example that needs this." is the wrong way around. The MSHub object is the real thing. That is what LEGO is using in their app. That is what LEGO is documenting. The hub object is the hidden "driver" that MSHub is built on. We are not supposed to use hub directly as be aware that using APIs at that level may have cause odd behavior

This is a screenshot when starting an empty Python program in the LEGO app, with HELP opened.

[harbaum]

LEGO and their app are using MicroPython and a lot of custom python code on top. So when using the LEGO app you are not directly dealing with a bare MicroPython. Instead you deal with the LEGO python code on top. The LEGO app communicates with a Python program running under Python. It does not communicate with MicroPython itself.

µPIDE (like any other Micropython IDE) instead runs directly on MicroPython. It does not rely on any further Python code present on the device. µPIDE can work with any MicroPython device. The LEGO app can only work with the LEGO spike.

The "weird" stuff you see when logging into the Hub via Hyperterm or the like is actually some sensor output being sent by the still running LEGO Python code. Once you press ctrl-c you stop that. µPIDE (again like any other MicroPython IDE) also sends ctrl-c to stop any running code and to interact itself with MicroPython.

So the import/recursion problem arises somehow due to the LEGO code being stopped, other commands being executed and finnally custom user code to be run again. I will give other IDEs a try this evening. But since I am using the official pyboard.py code to communicate with MicroPython I'd assume that other IDEs using the same code face the same problems.

I could soft reboot the Hub (e.g. send ctrl-d) before I run custom programs. That may solve the LEGO issue but will slow the whole interaction down significantly as the reboot takes some time.

[harbaum]

This is something that is present somewhere inside the pyboard.py. This is what happens when I do a "ampy run import.py" with the import you posted. Ampy is a command line tool made by adafruit which has nothing to do with µPIDE:

$ ampy -p /dev/ttyACM3 run import.py 
Traceback (most recent call last):
...
ampy.pyboard.PyboardError: ('exception', b'', b'Traceback (most recent call last):\r\n  File "<stdin>", line 2, in <module>\r\n  File "mindstorms/__init__.py", line 1, in <module>\r\n  File "_api/__init__.py", line 1, in <module>\r\n  File "_api/speaker.py", line 1, in <module>\r\n  File "system/__init__.py", line 1, in <module>\r\n  File "system/callbacks/__init__.py", line 1, in <module>\r\n  File "util/error_handler.py", line 1, in <module>\r\n  File "protocol/__init__.py", line 1, in <module>\r\n  File "protocol/rpc_protocol.py", line 1, in <module>\r\n  File "protocol/notifications.py", line 1, in <module>\r\nRuntimeError: maximum recursion depth exceeded\r\n')

[harbaum]

pyboard.py itself already triggers that:

$ python3 ./pyboard.py import.py 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "mindstorms/__init__.py", line 1, in <module>
  File "_api/__init__.py", line 1, in <module>
  File "_api/speaker.py", line 1, in <module>
  File "system/__init__.py", line 1, in <module>
  File "system/callbacks/__init__.py", line 1, in <module>
  File "util/error_handler.py", line 1, in <module>
  File "protocol/__init__.py", line 1, in <module>
  File "protocol/rpc_protocol.py", line 1, in <module>
  File "protocol/notifications.py", line 1, in <module>
RuntimeError: maximum recursion depth exceeded

[harbaum]

Ok, I understand what happens. The code in pyboard.py (which uPIDE uses) actually does reset the board before running any code. However, this will trigger the devices startup sequence. Now µPIDE (nor any other IDE) can know what happens during device boot and how long this takes. So it assumes that once the device reports that it has rebooted it can tell the device to execute the code the user wants to run.

The LEGO Hub reports that it does a soft reboot. But that doesn't mean it has fully booted yet. So µPIDE (and the other IDEs) interrupt the LEGO hubs boot process. And this half-booted state triggers the recursion.

Hmmm .... I need to think about that. I wonder what side effects I'd see if I totally disable the reboot and just use the device in the state it's in ...

[harbaum]

Please check if 1.1.3 fixes this problem for you:

https://github.com/harbaum/upide/releases/tag/v1.1.3

[maarten-pennings]

I was using V1.1.2a. I downloaded V1.1.3. When I run it, it is active for about 1 sec (so it was hard to screenshot), then disappears.

[harbaum]

The old version still works as expected? The new version just closes? I'll try to investigate. But that may take a few days as I don't have a windows machine here to test.

[harbaum]

Small bugfix, again under https://github.com/harbaum/upide/releases/tag/v1.1.3b

This should hopefully fix the instant close.

[maarten-pennings]

V1.1.3b doesn't stop, finds hub, and runs programs using hub. Good!

However, it does not run programs using MSHub.

In REPL:

>>> from mindstorms import MSHub
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mindstorms/__init__.py", line 1, in <module>
  File "_api/__init__.py", line 1, in <module>
  File "_api/lightmatrix.py", line 1, in <module>
  File "system/__init__.py", line 1, in <module>
  File "system/callbacks/__init__.py", line 1, in <module>
  File "util/error_handler.py", line 1, in <module>
ImportError: can't import name notify_error_event
>>> 

When I make a one line program:

MShub.py, line 1
ImportError: no module named 'mindstorms.MSHub'

So, no longer the maximum recursion depth exceeded, but two different errors!

[harbaum]

No problem here in neither case.

You have to give the hub the time to fully boot (2-3 seconds after the heart appears). If you start µPIDE too early and interrupt the boot process then you start seeing these errors (again). Looks like LEGO didn't really care to handle exceptions and errors.

[maarten-pennings]

Check! I waited a couple of seconds after the heart appears and both REPL and a source file accepted the import MSHub without errors.

By the way, I have the Robot Inventor (not Spike Prime). Robot inventor has a different firmware than Spike Prime; it shows a play button instead of a heart. Don't know if there are more substantial differences!

Thanks for all the work.

[maarten-pennings]

I added an MSHub example - see issue 4.

[harbaum]

Fixed

[aivarannamaa]

It looks like LEGO has only tested importing mindstorms when _api was already imported (via hub_runtime). Otherwise the chain of dependent modules to import becomes longer than the recursion limit.

The work-around is to import _api before importing mindstorms.