FieldFriend rb26 not working as intended - Input delay when steered manually

[BenGeissler]

Hello,

When working on a FieldFriend robot of an older version - Not sure if its a version label, but it says zauberzeug@rb26 in the terminal - the robots manual steering is very delayed. I.e. the robot starts driving after having released Shift+ArrowUp. It drives at a fixed speed that is not affected by changing the speed value in the nicegui. It also does not seem to brake and instead rolls for about another meter slowing down to a stand still. When using the Joystick it is also very delayed.
When I first had this issue I was able to control the robot through the android App, which worked perfectly fine.
After a while, though, the device could not be found anymore and the driving directions forward/backward have inverted. So it drives backwards when it should drive forwards.

Issues are:

• Delayed and faulty reaction to Movement Inputs
• Not able to find the robot via Bluetooth in the Android App
• The whole NiceGui is quite slow to react to any button presses
• No image detected where the output of the camera looking down at the ground should

I can print you multiple outputs of ./docker.sh l rosys, because I had another issue with the nicegui not working until commented out a few lines of code.

The first Error Output:

Details

```html rosys_1 | AttributeError: 'NoneType' object has no attribute 'name' rosys_1 | 2024-04-15 15:36:48.277 [INFO] rosys/vision/detector_hardware.py:45: trying reconnect 26e999ba-9ee4-4940-af33-31dff177bf09 rosys_1 | 2024-04-15 15:36:48.277 [INFO] rosys/vision/detector_hardware.py:57: connecting to detector at ws://localhost:8004 rosys_1 | 2024-04-15 15:36:48.280 [INFO] rosys/vision/detector_hardware.py:45: trying reconnect 20792983-3cd7-4178-9f15-ec1925bec868 rosys_1 | 2024-04-15 15:36:48.281 [INFO] rosys/vision/detector_hardware.py:57: connecting to detector at ws://localhost:8005 rosys_1 | 2024-04-15 15:36:48.402 [INFO] rosys/vision/detector_hardware.py:59: connected successfully rosys_1 | 2024-04-15 15:36:48.403 [INFO] rosys/vision/detector_hardware.py:59: connected successfully rosys_1 | 2024-04-15 15:36:48.457 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:48.861 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:49.462 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:49.866 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:50.303 [INFO] field_friend/navigation/gnss.py:123: Found port: /dev/ttyACM1 - Septentrio USB Device - CDC Abstract Control Model (ACM) rosys_1 | 2024-04-15 15:36:50.304 [INFO] field_friend/navigation/gnss.py:125: Found GNSS device: /dev/ttyACM1 rosys_1 | 2024-04-15 15:36:50.305 [INFO] field_friend/navigation/gnss.py:133: Connecting to GNSS device "/dev/ttyACM1"... rosys_1 | 2024-04-15 15:36:50.306 [INFO] field_friend/navigation/gnss.py:139: Connected to GNSS device "/dev/ttyACM1" rosys_1 | 2024-04-15 15:36:50.711 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:50.974 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:51.002 [INFO] rosys/rosys.py:69: Enabling ESP: done rosys_1 | 2024-04-15 15:36:51.819 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:52.280 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:52.860 [INFO] field_friend/vision/camera_configurator.py:26: updating camera config rosys_1 | 2024-04-15 15:36:52.869 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:53.293 [WARNING] rosys/hardware/lizard_firmware.py:69: Could not read Lizard version from P0 rosys_1 | 2024-04-15 15:36:53.864 [ERROR] nicegui/nicegui.py:159: 'NoneType' object has no attribute 'name' rosys_1 | + Exception Group Traceback (most recent call last): rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups rosys_1 | | yield rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 190, in __call__ rosys_1 | | async with anyio.create_task_group() as task_group: rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__ rosys_1 | | raise BaseExceptionGroup( rosys_1 | | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) rosys_1 | +-+---------------- 1 ---------------- rosys_1 | | Traceback (most recent call last): rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ rosys_1 | | await self.app(scope, receive, _send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__ rosys_1 | | with collapse_excgroups(): rosys_1 | | File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__ rosys_1 | | self.gen.throw(typ, value, traceback) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups rosys_1 | | raise exc rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__ rosys_1 | | response = await self.dispatch_func(request, call_next) rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/app/nicegui/middlewares.py", line 10, in dispatch rosys_1 | | response = await call_next(request) rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next rosys_1 | | raise app_exc rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro rosys_1 | | await self.app(scope, receive_or_disconnect, send_no_error) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 24, in __call__ rosys_1 | | await responder(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 44, in __call__ rosys_1 | | await self.app(scope, receive, self.send_with_gzip) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ rosys_1 | | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app rosys_1 | | raise exc rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app rosys_1 | | await app(scope, receive, sender) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ rosys_1 | | await self.middleware_stack(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app rosys_1 | | await route.handle(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle rosys_1 | | await self.app(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app rosys_1 | | await wrap_app_handling_exceptions(app, request)(scope, receive, send) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app rosys_1 | | raise exc rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app rosys_1 | | await app(scope, receive, sender) rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app rosys_1 | | response = await func(request) rosys_1 | | ^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app rosys_1 | | raise e rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app rosys_1 | | raw_response = await run_endpoint_function( rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function rosys_1 | | return await dependant.call(**values) rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/app/nicegui/page.py", line 104, in decorated rosys_1 | | result = func(*dec_args, **dec_kwargs) rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | File "/app/field_friend/interface/pages/main_page.py", line 19, in page rosys_1 | | self.content(devmode=False) rosys_1 | | File "/app/field_friend/interface/pages/main_page.py", line 34, in content rosys_1 | | operation(self.system, leaflet_map_landing) rosys_1 | | File "/app/field_friend/interface/components/operation.py", line 134, in __init__ rosys_1 | | ui.label(f'Do you want to continue the canceled {"mowing" if self.automations_toggle.value == "mowing" else f"weeding on {self.system.weeding.current_row.name}"}?').classes( rosys_1 | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | | AttributeError: 'NoneType' object has no attribute 'name' rosys_1 | +------------------------------------ rosys_1 | rosys_1 | During handling of the above exception, another exception occurred: rosys_1 | rosys_1 | Traceback (most recent call last): rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ rosys_1 | await self.app(scope, receive, _send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__ rosys_1 | with collapse_excgroups(): rosys_1 | File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__ rosys_1 | self.gen.throw(typ, value, traceback) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups rosys_1 | raise exc rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__ rosys_1 | response = await self.dispatch_func(request, call_next) rosys_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/app/nicegui/middlewares.py", line 10, in dispatch rosys_1 | response = await call_next(request) rosys_1 | ^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next rosys_1 | raise app_exc rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro rosys_1 | await self.app(scope, receive_or_disconnect, send_no_error) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 24, in __call__ rosys_1 | await responder(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 44, in __call__ rosys_1 | await self.app(scope, receive, self.send_with_gzip) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ rosys_1 | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app rosys_1 | raise exc rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app rosys_1 | await app(scope, receive, sender) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ rosys_1 | await self.middleware_stack(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app rosys_1 | await route.handle(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle rosys_1 | await self.app(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app rosys_1 | await wrap_app_handling_exceptions(app, request)(scope, receive, send) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app rosys_1 | raise exc rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app rosys_1 | await app(scope, receive, sender) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app rosys_1 | response = await func(request) rosys_1 | ^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app rosys_1 | raise e rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app rosys_1 | raw_response = await run_endpoint_function( rosys_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function rosys_1 | return await dependant.call(**values) rosys_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/app/nicegui/page.py", line 104, in decorated rosys_1 | result = func(*dec_args, **dec_kwargs) rosys_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rosys_1 | File "/app/field_friend/interface/pages/main_page.py", line 19, in page rosys_1 | self.content(devmode=False) rosys_1 | File "/app/field_friend/interface/pages/main_page.py", line 34, in content rosys_1 | operation(self.system, leaflet_map_landing) rosys_1 | File "/app/field_friend/interface/components/operation.py", line 134, in __init__ rosys_1 | ui.label(f'Do you want to continue the canceled {"mowing" if self.automations_toggle.value == "mowing" else f"weeding on {self.system.weeding.current_row.name}"}?').classes( rosys_1 ```

After commenting out the following code in field_friend/components/operation.py lines 133-139, the nicegui was working.

        # with ui.dialog() as self.dialog, ui.card():
        #     ui.label(f'Do you want to continue the canceled {"mowing"  if self.automations_toggle.value == "mowing" else f"weeding on {self.system.weeding.current_row.name}"}?').classes(
        #         'text-lg')
        #     with ui.row():
        #         ui.button('Yes', on_click=lambda: self.dialog.submit('Yes'))
        #         ui.button('No', on_click=lambda: self.dialog.submit('No'))
        #         ui.button('Cancel', on_click=lambda: self.dialog.submit('Cancel'))

Second Error output:

Details

```html rosys_1 | IndexError: pop from empty list rosys_1 | 2024-04-15 16:48:16.741 [ERROR] rosys/rosys.py:206: error in "RobotHardware.update" rosys_1 | Traceback (most recent call last): rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/rosys/rosys.py", line 200, in _repeat_one_handler rosys_1 | await invoke(handler) rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/rosys/helpers.py", line 18, in invoke rosys_1 | result = await result rosys_1 | ^^^^^^^^^^^^ rosys_1 | File "/home/zauberzeug/.local/lib/python3.11/site-packages/rosys/hardware/robot.py", line 66, in update rosys_1 | cast(ModuleHardware, module).handle_core_output(time, words) rosys_1 | File "/app/field_friend/hardware/y_axis.py", line 201, in handle_core_output rosys_1 | self.end_l = int(words.pop(0)) == 0 rosys_1 | ^^^^^^^^^^^^ rosys_1 | Error grabbing logs: unexpected EOF ```

So far i tried removing the rosys and nicegui folders in the field_friend directory and starting the dockers with ./docker.sh U. As well as going around in the code lines indicated in the error outputs, which did not help so I reset my own changes to the code.

I was not able to figure out why this delay is happening or how to fix it. If anyone could help me with this it would be highly appreciated.

[rodja]

That is very strange behaviour indeed @BenGeissler. I suggest we first help you to upgrade to the latest software and then investigate the problems -- if they are still present afterwards.

Please have a look at our deployment instructions in the README. Ideally you move the existing field_friend, rosys and nicegui folders to a subfolder old or similar and then execute all steps of the Setup (including the two optional ones).

[BenGeissler]

Hello, first of all thank you for your quick answer.

TLDR; The robot can be smoothly steered only via the android app again. Flashing the Core and P0 did the trick

I have followed the detailed instructions of the README. The point regarding the updating of the lizard firmware did the trick. I even was so highly motivated that I misunderstood the task and locally installed/updated the lizard directory by hand by following the instructions of lizard#development. Where I also want to take the opportunity to let you know, that there is no get_owl.sh file in the latest main branch of lizard. I found the file in the commit history of the main branch and cloned owl by hand.

After having a look into the Development section of the Web Interface, the robot can be steered again via phone and the driving directions also work as intended.

However, steering the robot via the Field Friend Web Interface still imposes a few bugs. The robot starts moving about half a second after a button was pressed. It also stops on its own after about 3 seconds of moving, while the button was continuously pressed. It behaves in the same way with inputs via the NiceGui Joystick.

Because of the movement being smooth via the bluetooth connection, I suspect the error to be somewhere within Nicegui. Which would also explain the slow reaction of any other button presses in the Web Interface.

[angelom93]

Hello @BenGeissler,

I think we've solved the problem. We changed the structure of the GNSS update to allow increased frequency. The issue was that the update function blocked the system for nearly one second while waiting for the next GNSS message.

Commit https://github.com/zauberzeug/field_friend/commit/33582091f3b2a684d1cb3fd101f1367b96bcab08 resolves this by running the serial reading in an IO-bound process.

[BenGeissler]

Hello,

this fixed my issue and the robot is now driving smoothly. The Web Interface is also running/reacting faster now.

Thank you for your quick feedback!