LEGO MINDSTORMS NXT Support

[dlech]

@laurensvalk, it's great that you already have the very same idea! The NXT hubs, which are actually quite powerful hardware, will be wasted without a modern, preferably Python, API. If I'm not wrong, the existing Python API for NXT is https://github.com/Eelviny/nxt-python, which is about 5 years old already and currently unmaintained.

What does it take, generally, to write the NXT hub module in PyBricks? How can we help? -- I can look among my friend circle for engineers who would like to take on this, with your guidance. Also, what sponsorship is needed for projects like this?

Originally posted by @TheVinhLuong102 in https://github.com/pybricks/support/issues/29#issuecomment-685858236

[dlech]

I think the first thing that needs to be done before getting into the code is to design the user experience. The NXT is quite different in this regard compared to the Powered Up hubs we are currently focused on.

• NXT has a screen and extra buttons. What does the user interface on the screen look like? Should we make it like the official LEGO firmware? Should we make it like the LeJOS firmware? What features should be included? What features should be excluded?
• NXT is Bluetooth Classic and USB. Both of these are not compatible with the web IDE (code.pybricks.com) which is currently Bluetooth Low Energy only. We need to find out what it would take to add these to the web IDE, if it is even technically possible or look at using a different IDE.
• What will need to be changed in the IDE to allow for the different connection methods? Can this be done in a way that is easy to understand for non-technical users? Can this be done in a way that doesn't detract from the Powered Up user's experience?

To answer these questions, what I would like to see is mock-ups, flow charts, etc. that give a very precise specification of how things should look and behave from the user's point of view.

Once we have a clear idea of what the end product should be, then we can start discussing specifics on how to get there.

[TheVinhLuong102]

@dlech yes the UX design should indeed be the first step. Could you point us to the good references for the above considerations, so that we can start reading up? There're lots of historical considerations and decisions that new members like myself need to understand.

[laurensvalk]

Once we have a clear idea of what the end product should be, then we can start discussing specifics on how to get there.

In fact, taking a step further back, there are more fundamental things to clear up before even defining what the product should be, such as:

• Who will use it?
• Why will they use it?
• What will they use it for?
• How will they use it?

Let's not get trapped in technicalities at this stage, there's plenty of time for that later.

[dlech]

This is a good read too (asks the same sorts of questions): https://mu.readthedocs.io/en/latest/user-experience.html

[TheVinhLuong102]

@laurensvalk some initial answers I can think of are as follows:

• Who will use it? Those who want exposure to sufficiently serious robotics but have limited budgets per person. This is an important issue in developing countries.

• Why will they use it? The sufficiently powerful hardware specs at much lower hardware cost. NXT sets are being let go at significantly cheaper prices, as hobbyists in developed countries upgrade to the EV3 and beyond.

• What will they use it for? Robotics and computing programming education at scale, e.g. at public schools with limited funding, or in non-profit programs.

• How will they use it? One of the experiences I can think of is programming in Python on cheap laptops such as Chromebooks, and sending the Python code to NXT-based bots through some IDE like the EV3Dev VSCode plug-in.

[GianCann]

@TheVinhLuong102 there are many low cost/cheap educational product, like Micro:bit or similar, that having also a integrated features like Bluetooth Low Energy, gyro, temperature and light sensor and that can be drive LEGO motors and can be integrate with LEGO pieces. Have a sense to work on Pybricks for run it on the (very old) NXT Brick? I'm not so sure, honestly...

[JakubVanek]

Hi,

I've accidentally created https://github.com/pybricks/support/issues/169 as I've missed this issue. I definitely agree that the UX should be solved first, I forgot about that in the first issue. I have another set of possible answers to the product definition:

• Who will use it? Students learning programming and robotics.
• Why will they use it? Because they can then use (?) the same program on the EV3 (the course has loads of NXTs and EV3s).
• What will they use it for? The first task is to write a line follower program, next task is to partake in a competition (the actual task varies, but it can be maze navigation, ball throwing, ...).
• How will they use it? They would write a program in an IDE (installable is not a problem), then they'd upload it to the brick and see it the robot does what it should do (+ some debugging utility needs to be there, be it display output or remote console).

With regards,

Jakub

[JakubVanek]

• NXT is Bluetooth Classic and USB. Both of these are not compatible with the web IDE (code.pybricks.com) which is currently Bluetooth Low Energy only. We need to find out what it would take to add these to the web IDE, if it is even technically possible or look at using a different IDE.

The USB part could perhaps be handled by coding both sides to be using WebUSB,. However, it may break the Python serial console and/or compatibility with other clients relying on the NXT/Fantom driver. Bluetooth might be harder, I don't know much about it.

[laurensvalk]

Related: There's a fairly high percentage of NXT bricks whose screens no longer work after all these years. While there are ways to repair it, I think a more appealing solution might be to make Pybricks for NXT work well even without a screen. Perhaps by making the interface very simple, such as running/saving only one program like we do on most hubs today. There's no status light unfortunately, but there is the speaker for beeps.

[laurensvalk]

The USB part could perhaps be handled by coding both sides to be using WebUSB,. However, it may break the Python serial console and/or compatibility with other clients relying on the NXT/Fantom driver. Bluetooth might be harder, I don't know much about it.

Would USB-only be sufficient?

[JakubVanek]

Would USB-only be sufficient?

For our main use case the answer is yes, although I can see that for some more advanced projects brick-to-brick or brick-to-PC communication via Bluetooth could be beneficial. However, these could always be added later.

[laurensvalk]

Here's a very minimal but possibly viable user experience. It's a very bare-bones solution, which may have to be done regardless of the choices above.

• No screen interface/menu.
• Only one program saved on the brick. Start/stop it with the orange button.
• USB serial used for downloading new program, and standard in/out. No Bluetooth.
• No IDE, just a command line tool to upload the program

This is very similar to the experience we have already for Prime Hub / Inventor Hub for now, so there is plenty of inspiration, and it would mainly be a matter of writing equivalent low-level drivers for the right hardware.

This essentially avoids the user experience question for now, but it could be built on top of this at a later stage. As discussed above, we would normally define it first to ensure we don't lose focus on Powered Up, but we wouldn't want to discourage contributors who are more interested in NXT from taking this on if they want to. And going for a generic minimal solution like this first would certainly be interesting.

@JakubVanek, @TheVinhLuong102, what do you think?

[JakubVanek]

• No screen interface/menu.
• Only one program saved on the brick. Start/stop it with the orange button.
• USB serial used for downloading new program, and standard in/out. No Bluetooth.
• No IDE, just a command line tool to upload the program

I agree, this looks reasonable for bringing up the platform. GUI can be added later and it can probably be shared between multiple platforms (at minimum the EV3RT port could benefit from it). As such, it would be necessary to reach wider consensus on its design.

[TheVinhLuong102]

@laurensvalk @JakubVanek thank you for continuing this discussion. Yes, I agree with @laurensvalk's definition of the minimum-viable experience above. One question though: in the cases where the screen is broken, how can the user know whether the program has been downloaded to the brick or not, and how can the user select it to start?

[laurensvalk]

how can the user select it to start?

Using the center button.

how can the user know whether the program has been downloaded to the brick or not

You can't "see" it, but it's just always the most recent program. Most of the modern hubs don't have a screen either and just a single button, and it's not too bad.

Those hubs do have a colored light to show you the current status, but maybe we can do that with beeps on the NXT.

We can still make it show a status on the screen, which is nice when the screen is still working, but just make sure it is still possible to use without it.

[TheVinhLuong102]

@laurensvalk great, I think the proposed minimum-viable experience is appropriate. 👍

[Salva-OA]

NXT 2 can be used as an open hardware device which is explained in some examples here: http://www.nxtorm.es/menu-sensores-analogicos-caseros-nxt.html or buy here: http://www.mindsensors.com/37-ev3-and-nxt

Also, we could adapt previous interfaces like Mathlab, Lejos, scratch 2: https://codigo21.educacion.navarra.es/autoaprendizaje/primeros-pasos-con-enchanting-y-lego-nxt/ into pybricks and maintain school material longer, ... it is important to learn how a sensor or actuator behaves and how to approach a solution to a problem just with the tools you have on your hands, rather than have the latest "gadget". 300 € to 700 € x 20 students can be too much money for continuous updates. Later if the student has the opportunity to use modern sets like "spike prime" or micro:bit or even vice versa the learning process can be very smooth.

Could also be a good opportunity to re-join and publish good books ideas like "The LEGO MINDSTORMS NXT 2.0 Discovery Book" (Laurens), 150 proyectos (Ernesto Martínez de Carvajal Hedrich), "Creating Cool MINDSTORMS NXT Robots (Benedettelli)". just for the fun of rebuilding the models and moving from icon flow code to text (python)

Material that has not been re-manufactured like the solar and pneumatics sets can be used again, like these examples http://blog.electricbricks.com/2010/10/energia-eolica/ https://education.lego.com/es-es/lessons/ev3-science/acceleration-of-gravity

About the questionary Who will use it? Lego and electronic adult hobbyists who own a Mindstorms set or can buy it (Wallapop, eBay, ...) Schools and universities that have these sets and can create robotic clubs, or complement learning concepts with a physical experiment, mean more focus on hardware than code skills.

Why will they use it? Only because they own it and with the proper cable adaptors plus electronic, coding knowledge can enhance their "robot" builds. Probably if they do not have any robotics set, they will buy an arduino/esp32 compatible board set or a spike-prime set

What will they use it for? I think only they use pybricks if it allows this "open" approach, which is a small "kernel-firmware"; a document explaining the pins of the ports: analog, digital, ic2, uart; the ways to connect "official Lego peripherals", micro bit peripherals, and custom made, protocols ... and later how to model these in code: objects, libraries, files, folders ... etc

How will they use it? A copy of this interface https://python.microbit.org/v/3 could be nice since it allows you to make a simulation of behavior without having the microcontroller connected and also can be non-system operating dependent (several Spanish schools in Spain use Linux), but maybe for creating your own peripherical file library probably a local IDE for Linux, Windows would be better. If with time the LCD is broken, could be a great idea to generate an emulator, act as a program selector, and debug console.