[Question] Question with Pybricks on the EV3 (occasional program lock up)

[badassloumd]

Question Hello. Thank you for all of your efforts with the EV3 and Pybricks. You have done a great deal to inspire the next generation of scientists / programmers and engineers with the fantastic work you have done. My FLL team went from zero to Python Hero's in 6 short months!

I have been running into an issue with the EV3 / Pybricks. On rare occasion, as a motor is commanded to go - the ev3 / motor just stops running the code with out any explanation - it just hangs. It requires a stopping of the python program (via pressing the stop button on the brick) This is an intermittent failure, say 1 out of every 500 motor commands, but it is certainly there. If I had to guess, I would say it is an issue with a race condition in some code in the device driver that causes a reset - but I am guessing - no proof.

I have spent months trying to catch this elusive bug. It does not seem to be dependent on battery voltage, a particular motor, or even a particular brick, or a particular cable. I have tried changing the acceleration rates of the motor(s) to no avail. I have even put tiny bypass capacitors across the motor windings to help eliminate EMI. No joy, the intermittent fault still happens.

I am somewhat skilled in Linux, (and very tenacious at finding problems) and I often SSH in to the brick. Is there a log file I can look at that may give me / us some insight as to what the cause of this intermittent failure may be? Is there a log file for the motor device drivers?

Again, Thank you for all you have done.

Context What are you trying to accomplish that led to this question? Help fix Pybricks code locks up on EV3 at rare, occasional intervals

Screenshots There is a saying that a picture is worth a 1000 words. Screenshots really help to identify and solve problems.

[dlech]

Possibly related to #24. We've never been able to track this down since it happens so rarely.

You can run dmesg on the EV3 to see the kernel logs.

You could also try running pybricks-micropython with strace (you might have to sudo apt update && sudo apt install strace if it isn't installed by default). This will let us see all system calls the program makes (but is also really slows down the program). If you can happen to capture a faulty run with this, then we could compare it to a good run to see if there is some kind of race condition like you suspect.

[badassloumd]

Will do - I am going to run it from a power supply too to further eliminate the battery voltage variable

[badassloumd]

How do I start micro python with star trace?

On Oct 20, 2022, at 12:22 PM, David Lechner @.***> wrote:

 Possibly related to #24. We've never been able to track this down since it happens so rarely.

You can run dmesg on the EV3 to see the kernel logs.

You could also try running pybricks-micropython with strace (you might have to sudo apt update && sudo apt install strace if it isn't installed by default). This will let us see all system calls the program makes (but is also really slows down the program). If you can happen to capture a faulty run with this, then we could compare it to a good run to see if there is some kind of race condition like you suspect.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[dlech]

 brickrun -r strace pybricks-micropython my_file.py

[badassloumd]

OK will try it tomorrow AM!! Thank you

Also I ran DMESG and got some weird out-of-space error -- Like it over ran itself and then quit. I will look into this and try and see why

Thank you!!

---Lou

On 2022-10-20 20:52, David Lechner wrote:

brickrun -r strace pybricks-micropython my_file.py

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you authored the thread.Message ID: @.***>

Links:

[1] https://github.com/pybricks/support/issues/742#issuecomment-1286355333 [2] https://github.com/notifications/unsubscribe-auth/AOU7DDGHWEEYPUHX2JBVRSTWEHZO5ANCNFSM6AAAAAARKKAGF4

[badassloumd]

Interesting... When I run the code with strace the motors swing wildly out of control. I wonder if the added delay is somehow messing with the control loop that runs the motors?

On 2022-10-20 20:52, David Lechner wrote:

brickrun -r strace pybricks-micropython my_file.py

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you authored the thread.Message ID: @.***>

Links:

[1] https://github.com/pybricks/support/issues/742#issuecomment-1286355333 [2] https://github.com/notifications/unsubscribe-auth/AOU7DDGHWEEYPUHX2JBVRSTWEHZO5ANCNFSM6AAAAAARKKAGF4

[dlech]

Yes, it adds major delays. Maybe there is a way to make a simpler test case that just starts and stops a motor repeatedly without worrying about speed control?

[badassloumd]

Uggh,, so the real time control of the motors is done in the foreground as a Linux process? I thought for sure that the motion control would be run in one of the PRU's on that CPU.

Without using strace I am still seing

some strange stuff in dmesg, specifically "serial 18250: too much work for irq 53" but to me, that sounds like an over-loaded uart.

Do you know where I can get a copy of the source code for the code that runs the motors? Actually, a more basic question is how does micropython "talk" to devices like motor's?

for example, I see the directory tree:

/sys/devices/platform/ev3-ports/ev3-portsout:A/port4/ there are files in there, for example the "status" file that reflects weather or not a motor is pugged in. I assume that the motor can be interacted with from this interface. Do you know where I can find out more information on how to do this directly?

Thank you!

---Lou

On 2022-10-21 10:07, David Lechner wrote:

Yes, it adds major delays. Maybe there is a way to make a simpler test case that just starts and stops a motor repeatedly without worrying about speed control?

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you authored the thread.Message ID: @.***>

Links:

[1] https://github.com/pybricks/support/issues/742#issuecomment-1287093393 [2] https://github.com/notifications/unsubscribe-auth/AOU7DDCDEBOTSAZEGFR5Q33WEKWSFANCNFSM6AAAAAARKKAGF4

[dlech]

Since ev3dev is Linux where "everything is a file", motors are controlled by writing to sysfs attributes, i.e. in /sys/class/tacho-motor/motor0 (this is a symlink, so the actual Pybricks code may use a different path). You can play around with the from the command line by using cat ... to read attributes (files) and echo "xyz" > ... to write attributes (files).

Pybricks MicroPython uses it's own motor control algorithms instead of the ones from the ev3dev kernel driver. So we just use the run-direct command and write to the duty_cycle attribute and ignore everything else.

The code for Pybricks MicroPython v2.x that ships with ev3dev can be found at https://github.com/pybricks/pybricks-micropython/tree/v2. The control algorithms are in lib/pbio/src and the lower level code for accessing the sysfs attributes is in lib/pbio/drv/ev3dev_stretch and lib/ev3dev.

[badassloumd]

OK so I know enough linux to be dangerous. To make the motors do you write to /sys/class/tacho-motor/motor0/ There are a whole bunch of files there, for example

position position_sp

Where can I find out which one to write to to make the motor go to a particular position?

Thanks for all the help - I promise

to find this problem!!

---Lou

On 2022-10-21 12:31, David Lechner wrote:

Since ev3dev is Linux where "everything is a file", motors are controlled by writing to sysfs attributes, i.e. in /sys/class/tacho-motor/motor0 (this is a symlink, so the actual Pybricks code may use a different path). You can play around with the from the command line by using cat ... to read attributes (files) and echo "xyz" > ... to write attributes (files).

Pybricks MicroPython uses it's own motor control algorithms instead of the ones from the ev3dev kernel driver. So we just use the run-direct command and write to the duty_cycle attribute and ignore everything else.

The code for Pybricks MicroPython v2.x that ships with ev3dev can be found at https://github.com/pybricks/pybricks-micropython/tree/v2. The control algorithms are in lib/pbio/src and the lower level code for accessing the sysfs attributes is in lib/pbio/drv/ev3dev_stretch and lib/ev3dev.

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you authored the thread.Message ID: @.***>

Links:

[1] https://github.com/pybricks/support/issues/742#issuecomment-1287249582 [2] https://github.com/notifications/unsubscribe-auth/AOU7DDA67LEFU5SEFZJX6ETWELHOFANCNFSM6AAAAAARKKAGF4

[dlech]

You can read all about it at https://docs.ev3dev.org/projects/lego-linux-drivers/en/ev3dev-stretch/motors.html#tacho-motor-subsystem.

[badassloumd]

Hello. Its been a while since we chatted, I hope you are well.

I think I may have found a bug with Pybricks. I have a work around, but you might find this interesting.

I wrote a piece of ode that "rotates" a 2-wheeld robot. The code works (I did a bit of testing with it) but I later found a very strange condition that can rarely occur, where the left-motor is given a positive target rotation, and the right motor given the negative, (but same absolute value) rotation , -- however both motors will turn in the negative direction.

In this case, the value was -76.xxx degrees on

one motor and 76.xxx degrees on the other.

Again the odd thing is that the values were printed to my console as negative and positive, however both motors were turning backwards, and this only happened to fail on occasion, not every tine / always: Here is a code snippet of what --sometimes-- did not work:

---

     lmd =  how_many_tire_degress
     rmd =  -1*how_many_tire_degress

     print  ("LMD=",lmd," RMD=",rmd," Desired angle =",angle," 

turn_speed=",turn_speed," error =",error," Angle= " , actual_angle, "how_many_tire_degress =",how_many_tire_degress)

     left_motor.run_target(turn_speed,lmd,wait=False)
     right_motor.run_target(turn_speed,rmd,wait=True)
     actual_angle = gyro.angle()
     error = (actual_angle-angle)
     right_motor.reset_angle(0)
     left_motor.reset_angle(0)

---

This is what I did to fix the problem:

     lmd = int (how_many_tire_degress)
     rmd = int(-1*how_many_tire_degress)

     print  ("LMD=",lmd," RMD=",rmd," Desired angle =",angle," 

turn_speed=",turn_speed," error =",error," Angle= " , actual_angle, "how_many_tire_degress =",how_many_tire_degress)

     right_motor.reset_angle(0)
     left_motor.reset_angle(0)
     left_motor.run_target(turn_speed,lmd,wait=False)
     right_motor.run_target(turn_speed,rmd,wait=True)
     actual_angle = gyro.angle()
     error = (actual_angle-angle)

---

I have sent you the entire robot source code in case you need it, hopefully this will be helpful. I can repeat the error, and if you like take some logs etc..

 ---Lou