search

ev3dev / ev3dev-lang-python

Pure python bindings for ev3dev
MIT License
422 stars 146 forks source link

Threading error when 2 Threads may control the same Motor. Errors encountered in EV3Dev2 with different error messages on MicroPython vs Python3. No such errors encountered in EV3Dev(1) and PyBricks. #750

Open AntoniLuongPham opened 4 years ago

AntoniLuongPham commented 4 years ago

ev3dev version: 4.14.117-ev3dev-2.3.5-ev3

I have certain programs that crash on EV3Dev2 when 2 parallel Threads may control the same Motor. The crashes have different error messages on MicroPython vs. Python3. I have reproduced the errors in the enclosed test scripts "BUG.SameMotor.Threading.EV3Dev2.MicroPython.py" and "BUG.SameMotor.Threading.EV3Dev2.Python3.py".

For comparison, I have also tried and enclosed below other variations (e.g. using ev3dev(1.2.0) or pybricks instead of ev3dev2, or using multiprocessing or PyBricks's run_parallel instead of threading) that run successfully.

All these programs do the following: turn the Motor clockwise when the Touch Sensor is pressed, or turn the same Motor counter-clockwise when any IR Remote Control button is pressed. The observations from all the coding variations are as follows:

  1. threading with ev3dev2 2.1.0 on micropython: CRASHED after pressing the IR Remote Control buttons a few times (making the Motor turn counter-clockwise), even if the Touch Sensor is NOT pressed at all (i.e. that Touch-Sensor-triggered Thread never commands the Motor at all).

    File "ev3dev2/motor.py", line 1048, in on_for_seconds File "ev3dev2/motor.py", line 928, in wait_until_not_moving File "ev3dev2/motor.py", line 908, in wait OSError: 4

  2. threading with ev3dev2 2.1.0 on python3: CRASHED when the Touch Sensor and an IR Remote Control button are pressed at the same time, i.e. both Thread are trying to command the Motor at the same time. But the program runs ok when only 1 button is pressed.

    File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 1048, in on_for_seconds self.wait_until_not_moving() File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 928, in wait_until_not_moving return self.wait(lambda state: self.STATE_RUNNING not in state or self.STATE_STALLED in state, timeout) File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 908, in wait self._poll.poll(poll_tm) RuntimeError: concurrent poll() invocation

  3. multiprocessing with ev3dev2 2.1.0 on micropython: both Processes run successfully WITH mutual blocking, i.e. 1 Process CANNOT interrupt/reverse the other Process's Motor movement mid-stream

  4. multiprocessing with ev3dev2 2.1.0 on python3: both Processes run successfully WITH mutual blocking, i.e. 1 Process CANNOT interrupt/reverse the other Process's Motor movement mid-stream

  5. threading with ev3dev (1.2.0) on python3: both Threads run successfully WITHOUT mutual blocking, i.e. 1 Thread CAN interrupt/reverse the other Thread's Motor movement mid-stream

  6. multiprocessing with ev3dev (1.2.0) on python3: both Processes run successfully WITHOUT mutual blocking, i.e. 1 Process CAN interrupt/reverse the other Process's Motor movement mid-stream

  7. threading with pybricks (2.0.0) on pybricks-micropython: both Threads run successfully WITH mutual blocking, i.e. 1 Thread CANNOT interrupt/reverse the other Thread's Motor movement mid-stream

  8. run_parallel (experimental PyBricks feature) with pybricks (2.0.0) on pybricks-micropython: both Threads run successfully WITH mutual blocking, i.e. 1 Thread CANNOT interrupt/reverse the other Thread's Motor movement mid-stream

AntoniLuongPham commented 4 years ago

BUG.SameMotor.Threading.EV3Dev2.MicroPython.py

#!/usr/bin/env micropython

from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from threading import Thread

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)

Thread(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()
# *** BUG as of 2020 ***
# *** just the presence of the above parallel Thread makes this main thread crash ***
# Traceback (most recent call last):
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.MicroPython.py", line 38, in <module>
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.MicroPython.py", line 29, in press_any_ir_remote_button_to_turn_motor_counterclockwise
#   File "ev3dev2/motor.py", line 1048, in on_for_seconds
#   File "ev3dev2/motor.py", line 928, in wait_until_not_moving
#   File "ev3dev2/motor.py", line 908, in wait
# OSError: 4
AntoniLuongPham commented 4 years ago

BUG.SameMotor.Threading.EV3Dev2.Python3.py

#!/usr/bin/env python3

from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from threading import Thread

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)

Thread(target=touch_to_turn_motor_clockwise,
       daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# *** BUG as of 2020 ***
# *** as soon as both Touch Sensor and an IR button are pressed ***
# Traceback (most recent call last):
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.Python3.py", line 39, in <module>
#     press_any_ir_remote_button_to_turn_motor_counterclockwise()
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.Python3.py", line 33, in press_any_ir_remote_button_to_turn_motor_counterclockwise
#     block=True)
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 1048, in on_for_seconds
#     self.wait_until_not_moving()
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 928, in wait_until_not_moving
#     return self.wait(lambda state: self.STATE_RUNNING not in state or self.STATE_STALLED in state, timeout)
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 908, in wait
#     self._poll.poll(poll_tm)
# RuntimeError: concurrent poll() invocation
AntoniLuongPham commented 4 years ago

SameMotor.MultiProcessing.EV3Dev2.MicroPython.py

#!/usr/bin/env micropython

from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from multiprocessing import Process

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)

Process(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITH mutual blocking
# i.e. 1 process CANNOT interrupt/reverse the other process's Motor movement mid-stream
AntoniLuongPham commented 4 years ago

SameMotor.MultiProcessing.EV3Dev2.Python3.py

#!/usr/bin/env python3

from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from multiprocessing import Process

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)

Process(target=touch_to_turn_motor_clockwise,
        daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITH mutual blocking
# i.e. 1 process CANNOT interrupt/reverse the other process's Motor movement mid-stream
AntoniLuongPham commented 4 years ago

SameMotor.Threading.EV3Dev1.py

#!/usr/bin/env python3

from ev3dev.ev3 import TouchSensor, INPUT_1, InfraredSensor, INPUT_4, RemoteControl, MediumMotor, OUTPUT_A

from threading import Thread

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
REMOTE_CONTROL = RemoteControl(sensor=InfraredSensor(address=INPUT_4),
                               channel=1)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.run_timed(
                speed_sp=1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if REMOTE_CONTROL.buttons_pressed:
            MOTOR.run_timed(
                speed_sp=-1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)

Thread(target=touch_to_turn_motor_clockwise,
       daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both threads run successfully WITHOUT mutual blocking
# i.e. 1 thread CAN interrupt/reverse the other thread's Motor movement mid-stream
AntoniLuongPham commented 4 years ago

SameMotor.MultiProcessing.EV3Dev1.py

#!/usr/bin/env python3

from ev3dev.ev3 import TouchSensor, INPUT_1, InfraredSensor, INPUT_4, RemoteControl, MediumMotor, OUTPUT_A

from multiprocessing import Process

TOUCH_SENSOR = TouchSensor(address=INPUT_1)
REMOTE_CONTROL = RemoteControl(sensor=InfraredSensor(address=INPUT_4),
                               channel=1)
MOTOR = MediumMotor(address=OUTPUT_A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.run_timed(
                speed_sp=1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if REMOTE_CONTROL.buttons_pressed:
            MOTOR.run_timed(
                speed_sp=-1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)

Process(target=touch_to_turn_motor_clockwise,
        daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITHOUT any mutual blocking
# i.e. 1 process CAN interrupt/reverse the other process's Motor movement mid-stream
AntoniLuongPham commented 4 years ago

SameMotor.Threading.PyBricks.py

#!/usr/bin/env pybricks-micropython

from pybricks.ev3devices import TouchSensor, InfraredSensor, Motor
from pybricks.parameters import Port, Stop

from threading import Thread

TOUCH_SENSOR = TouchSensor(port=Port.S1)
IR_SENSOR = InfraredSensor(port=Port.S4)
MOTOR = Motor(port=Port.A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.pressed():
            MOTOR.run_time(
                speed=1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons(channel=1):
            MOTOR.run_time(
                speed=-1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)

Thread(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both threads run successfully WITH mutual blocking
# i.e. 1 thread CANNOT interrupt/reverse the other thread's Motor movement mid-stream
AntoniLuongPham commented 4 years ago

SameMotor.RunParallel.PyBricks.py

#!/usr/bin/env pybricks-micropython

from pybricks.ev3devices import TouchSensor, InfraredSensor, Motor
from pybricks.parameters import Port, Stop

from pybricks.experimental import run_parallel

TOUCH_SENSOR = TouchSensor(port=Port.S1)
IR_SENSOR = InfraredSensor(port=Port.S4)
MOTOR = Motor(port=Port.A)

def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.pressed():
            MOTOR.run_time(
                speed=1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)

def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons(channel=1):
            MOTOR.run_time(
                speed=-1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)

run_parallel(
    touch_to_turn_motor_clockwise,
    press_any_ir_remote_button_to_turn_motor_counterclockwise)

# observation: both threads run successfully WITH mutual blocking
# i.e. 1 thread CANNOT interrupt/reverse the other thread's Motor movement mid-stream
WasabiFan commented 4 years ago

Yeah, multi-threaded access to the same motor is not (officially) supported, unfortunately. It's curious that your results seem to indicate that stock motors work un-modified with the v1.0 series; I am not sure why there's a difference there, probably due to blocking logic changes. It might be worth diffing the motor code to compare, but then again, since we haven't intentionally designed it to work with concurrent access I don't expect that to be consistent.

My suggestion is to go with the mutex version to be safe.

TheVinhLuong102 commented 4 years ago

@WasabiFan the most curious result among the above is from Program no. 1 "BUG.SameMotor.Threading.EV3Dev2.MicroPython.py": the program crashes even when the Touch Sensor is never touched and hence its related Thread never actually attempts to access the Motor. What do you make of this?

Is there a MicroPython compiler that compiles this program and identifies that there is a chance that the Touch-Sensor-related Thread may access the Motor?

WasabiFan commented 4 years ago

That case sounds like this other one: https://github.com/ev3dev/ev3dev-lang-python/issues/727

OSError 4 is EINTR, i.e. "system call interrupted". It means that a signal was raised while in a system call, and the system call was aborted to prevent deadlock. This happens because even just the presence of threads means the interpreter (Micropython specifically) will begin triggering signals to synchronize garbage collection.

Technical overview aside, it means that multithreading does not play nice with our MicroPython library, unfortunately. I had tried to work toward a true solution of this (starting with https://github.com/ev3dev/ev3dev-lang-python/pull/732, to benchmark the speed of our library before adding error handling) but haven't circled back to finish it, implement the error handling, and re-benchmark. There are some notes at the bottom of the thread linked above which might help you work around it.

foocross commented 3 years ago

We are also getting OSError: 4 errors in wait a significant number of times in micropython. If we shut off odometry_start() from MoveDifferential , would it remove the problem?

WasabiFan commented 3 years ago

Ah, yeah, odometry uses a thread in the background and may provoke EINTR.

foocross commented 3 years ago

Thanks