pypilot crashes when usb nmea stops data

[kagouraki]

I am trying Seans newest beta image of tinypilot tinypilot_beta_2024_01_14.img. I have a hat and motor controller from Sean, a pi zero w and the usb hub from waveshare ( https://www.waveshare.com/wiki/USB_HUBHAT(B) ) I have two of these boards connected to the hub and configured for USB to RS232 data ( https://www.aliexpress.com/item/1005004423540244.html )

everything seems to work perfectly until i stop the incoming nmea stream. Then it crashes. This is the terminal output :

tc@pypilot:~$ pypilot
autopilot start 172.770486
world magnetic model not available
ERROR loading wind.py cannot import name 'HeadingOffset'   cannot import name 'HeadingOffset'
ERROR loading learning.py liblzma.so.5: cannot open shared object file: No such file or directory   liblzma.so.5: cannot open shared object file: No such file or directory
ERROR loading intellect.py liblzma.so.5: cannot open shared object file: No such file or directory   liblzma.so.5: cannot open shared object file: No such file or directory
ERROR loading gps.py cannot import name 'HeadingOffset'   cannot import name 'HeadingOffset'
imu process 3319
made imu process realtime
Using settings file RTIMULib.ini
Settings file RTIMULib.ini loaded
Detected ICM20948 at standard address
Using fusion algorithm Kalman STATE4
IMU Name: ICM-20948
min/max compass calibration not in use
Using ellipsoid compass calibration
Using accel calibration
warning, failed to make calibration process idle, trying renice
nmea process 3327
ICM-20948 init complete
listening on port 20220 for nmea connections
IMU all sensor axes verified
gps process 3332
loading servo calibration /home/tc/.pypilot/servocalibration
WARNING: using default servo calibration!!
Available Pilots: ['basic', 'absolute']
warning: failed to open special file /dev/watchdog0 for writing
         cannot stroke the watchdog
pypilotServer process 3334
pypilotinit complete 176.201026
serialprobe found more devices by path
servo probe ('/dev/ttyAMA0', 38400) 176.286607
server/client is running too _slowly_ 0.11574099999998566
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800)
arduino servo found ['/dev/ttyAMA0', 38400]
serialprobe success: /home/tc/.pypilot/servodevice ['/dev/ttyAMA0', 38400]
Arduino motor controller EEPROM SIGNATURE ok
gpsd connected
GPSD devices []
load file /home/tc/.pypilot/pypilot.conf
store_file /home/tc/.pypilot/pypilot.conf.bak 178.135
server setup has 6 pipes
imu rate set to rate 20
server add socket ('127.0.0.1', 34444)
too long write pipe 0.10300588607788086 pypilotServer pipe3[1] 22
too long write pipe 0.05309104919433594 pypilotServer pipe3[1] 26
too long write pipe 0.0702662467956543 pypilotServer pipe0[1] 12
setting initial gyro bias [1.306, -0.351, -0.097]
too long write pipe 0.09571981430053711 pypilotServer pipe3[1] 42
imu rate set to rate 20
too long write pipe 0.14251017570495605 pypilotServer pipe3[1] 18
server add socket ('127.0.0.1', 34446)
too long write pipe 0.06312012672424316 pypilotServer pipe0[1] 113
sensors is running too _slowly_ 0.05821600000001581
signalk process 3331
sensors is running too _slowly_ 0.3469599999999957
signalkread token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXZpY2UiOiJweXBpbG90LTI1MTcwMjk3NDg5IiwiaWF0IjoxNjYyMTAzNjAzLCJleHAiOjE2OTM2NjEyMDN9.Hs_3V83weuk8KiDXCTZ59DMGS3rZEJCNfsU8W0IjnWA
zeroconf addresses ['127.0.0.1', '192.168.1.18'] 2
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 38400)
gps filter process 3333
nmea poll times 14.304 0.057 0.000 0.000 0.000 0.000 0.000 0.057
sensors is running too _slowly_ 0.05917400000001294
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 4800)
nmea poll times 19.479 0.057 0.000 0.000 0.000 0.000 0.000 0.057
sensors is running too _slowly_ 0.05908300000001532
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800)
serialprobe success: /home/tc/.pypilot/nmea0device ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800)
sensor found gps serial GP/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0 Wed Apr 12 11:03:08 2023
nmea poll times 20.351 0.001 0.209 0.002 0.000 0.000 0.000 0.213
sensors is running too _slowly_ 0.21474299999999857
sensor found water serial II/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0 Wed Apr 12 11:03:08 2023
sensor found wind serial WI/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0 Wed Apr 12 11:03:08 2023
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 38400)
calibration loaded, starting 3317
nmea poll times 33.015 0.057 0.003 0.000 0.000 0.000 0.000 0.060
sensors is running too _slowly_ 0.06194399999998268
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 4800)
nmea poll times 38.154 0.057 0.001 0.001 0.000 0.000 0.000 0.059
sensors is running too _slowly_ 0.06035099999999716
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 38400)
sensors is running too _slowly_ 0.02511599999999703
nmea poll times 59.505 0.057 0.001 0.000 0.000 0.000 0.000 0.059
sensors is running too _slowly_ 0.06198499999999285
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 4800)
nmea poll times 64.693 0.057 0.000 0.000 0.000 0.000 0.000 0.058
sensors is running too _slowly_ 0.05942600000000198
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 38400)
sensors is running too _slowly_ 0.02533299999998917
serial device dt 2.010177999999968 ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800) is another process accessing it?
serial device dt 2.0695280000000196 ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800) is another process accessing it?
serial device dt 2.131512000000015 ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800) is another process accessing it?
serial device dt 2.191433999999987 ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800) is another process accessing it?
serial device dt 2.2514819999999816 ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0', 4800) is another process accessing it?
nmea poll times 85.977 0.057 0.000 0.000 0.000 0.000 0.000 0.058
sensors is running too _slowly_ 0.060441000000025724
nmea probe ('/dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0', 4800)
sensors is running too _slowly_ 0.02603900000002568
sensor lost gps serial GP/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0 Wed Apr 12 11:04:14 2023
sensor lost water serial II/dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0 Wed Apr 12 11:04:14 2023
Traceback (most recent call last):
  File "/usr/local/bin/pypilot", line 11, in <module>
    load_entry_point('pypilot==0.44', 'console_scripts', 'pypilot')()
  File "/usr/local/lib/python3.6/site-packages/pypilot/autopilot.py", line 520, in main
    ap.iteration()
  File "/usr/local/lib/python3.6/site-packages/pypilot/autopilot.py", line 361, in iteration
    self.sensors.poll()
  File "/usr/local/lib/python3.6/site-packages/pypilot/sensors.py", line 402, in poll
    self.lostsensor(sensor)
  File "/usr/local/lib/python3.6/site-packages/pypilot/sensors.py", line 407, in lostsensor
    sensor.reset()
  File "/usr/local/lib/python3.6/site-packages/pypilot/sensors.py", line 361, in reset
    self.direction.set(False)
AttributeError: 'Water' object has no attribute 'direction'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pypilot/autopilot.py", line 188, in cleanup
    os.kill(pid, signal.SIGTERM) # get backtrace
  File "/usr/local/lib/python3.6/site-packages/pypilot/autopilot.py", line 194, in cleanup
    raise KeyboardInterrupt # to get backtrace on all processes
KeyboardInterrupt
closing autopilot
tc@pypilot:~$

The /dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0 has nothing connected so it keeps probing. Disregard it. Also i want to mention that i needed to change the 60-gpsd.rules and commented out the CP210x portion so it wont remove the hub after discovering it. I just made the exact same things in the tinypilot_2021_11_16.img.xz image and it works as expected.

I hope it is a easy fix because version 0.46 has awesome feautures !

[seandepagnier]

You have a water speed sensor correct? It tries to use this instead of gps speed to calculate true wind... but as you can see there is a bug if you remove the waterspeed sensor. It is great you found this, because most people dont have such a sensor. I have updated git to hopefully fix this.

You said "Also i want to mention that i needed to change the 60-gpsd.rules and commented out the CP210x portion so it wont remove the hub after discovering it."

I am not sure what you mean by "it wont remove the hub after discovering it" Could you explain a little more? I am assuming you have a cp210x serial device that is not a gps, but the rules think it is or?? I will try to figure a way to improve it if so.

[kagouraki]

ok, firsts things first. I installed a fresh tinypilot_beta_2024_01_14.img and tried to update to the main branch to test the changes you made for the water sensor:

cd
mkdir pypilot-update
cd pypilot-update/
git clone git@github.com:pypilot/pypilot.git
git clone --depth 1 git@github.com:pypilot/pypilot_data.git
cp -rv pypilot_data/* pypilot
cd pypilot
. pypilot.build
sudo reboot

after the reboot the screen is stuck at the logo so i ssh and try to run pypilot manually and get :

tc@pypilot:~$ pypilot
autopilot start 555.127886
falling back to python nonblocking socket, will consume more cpu cannot import name 'linebuffer'
pypilot failed to import required modules.  Did you forget to run sudo python3 setup.py install?
tc@pypilot:~$

when i run the sudo python3 setup.py install from the pypilot directory(the tinypilot image original one since i rebooted and the previously cloned main branch is deleted) the screen works and pypilot is started. In the info i see the version is 0.45. Also the web interface doent work.

I run the filestool.sh -b but after reboot the same thing. What is the procedure to update to the latest main branch from the image tinypilot_beta_2024_01_14.img ?

So as for the usb devices, I have the hardware mentioned previously. When i dont comment the cp210x line in the 60-gpsd.rules pypilot starts as this :

tc@pypilot:/mnt/mmcblk0p2/tinypilot/pypilot$ pypilot
autopilot start 1427.002011
world magnetic model not available
ERROR loading gps.py cannot import name 'HeadingOffset'   cannot import name 'HeadingOffset'
ERROR loading wind.py cannot import name 'HeadingOffset'   cannot import name 'HeadingOffset'
imu process 6697
made imu process realtime
Using settings file RTIMULib.ini
Settings file not found. Using defaults and creating settings file
Detected ICM20948 at standard address
Using fusion algorithm Kalman STATE4
IMU Name: ICM-20948
min/max compass calibration not in use
Using ellipsoid compass calibration
Using accel calibration
warning, failed to make calibration process idle, trying renice
ICM-20948 init complete
IMU all sensor axes verified
nmea process 6706
listening on port 20220 for nmea connections
gps process 6709
loading servo calibration /home/tc/.pypilot/servocalibration
WARNING: using default servo calibration!!
Available Pilots: ['absolute', 'basic']
warning: failed to open special file /dev/watchdog0 for writing
         cannot stroke the watchdog
pypilotServer process 6711
pypilotinit complete 1430.055879
serialprobe found more devices by path
serialprobe removing gps device /dev/serial/by-path/platform-20980000.usb-usb-0:1.4:1.0-port0
serialprobe removing gps device /dev/serial/by-path/platform-20980000.usb-usb-0:1.1:1.0-port0
servo probe ('/dev/ttyAMA0', 38400) 1430.152082
server/client is running too _slowly_ 0.13522699999998622
arduino servo found ['/dev/ttyAMA0', 38400]
serialprobe success: /home/tc/.pypilot/servodevice ['/dev/ttyAMA0', 38400]
Arduino motor controller EEPROM SIGNATURE ok
load file /home/tc/.pypilot/pypilot.conf
store_file /home/tc/.pypilot/pypilot.conf.bak 1431.150
server setup has 6 pipes
imu rate set to rate 20
too long write pipe 0.06209206581115723 pypilotServer pipe3[1] 15
gpsd connected
GPSD devices ['/dev/ttyUSB0', '/dev/ttyUSB1']
too long write pipe 0.07815408706665039 pypilotServer pipe3[1] 42
imu rate set to rate 20
setting initial gyro bias [0.763, -0.235, -0.047]
server add socket ('127.0.0.1', 34748)
server add socket ('127.0.0.1', 34750)
zeroconf addresses ['127.0.0.1', '192.168.1.18'] 2
sensors is running too _slowly_ 0.3948849999999311
signalk process 6708
signalkread token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXZpY2UiOiJweXBpbG90LTI1MTcwMjk3NDg5IiwiaWF0IjoxNjYyMTAzNjAzLCJleHAiOjE2OTM2NjEyMDN9.Hs_3V83weuk8KiDXCTZ59DMGS3rZEJCNfsU8W0IjnWA
gps filter process 6710
calibration loaded, starting 6696

and i get no nmea data because i understand it doesnt probe the input to recieve them. Truly i dont undestand the difference between gps device and nmea in device. Why should it know before recieving data what device it is? I am testing it on the bench and i have a nmeatester sendig this file over and over again with all sorts of nmea data. Nmea0183Tester.txt

[kagouraki]

Ok found the solution to the update procedure in the wiki in another section. You need to run the pypilot.build command twice before rebooting. Now with the main branch and version 0.45 the problem of the water sensor is fixed. But the web interface is not working anymore. The Hat configuration 33333 is working but the port 80 is saying

Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

error 500.

Should i open a new Issue ? The water sensor one is considered closed

[kagouraki]

I think here lies the web interface problem :

tc@pypilot:~$ pypilot_web 
using port 8000
failed to import flask_babel, translations not possible!! [Errno 2] No such file or directory: '/usr/local/lib/python3.6/site-packages/pypilot/web/translations'
processing clients
[2023-04-12 11:12:58,289] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/pypilot/web/web.py", line 197, in index
    return render_template('index.html', async_mode=socketio.async_mode, pypilot_web_port=pypilot_web_port, tinypilot=tinypilot.tinypilot, translations=translations, language=config['language'], languages=Markup(LANGUAGES))
NameError: name 'LANGUAGES' is not defined
[2023-04-12 11:12:58,711] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/pypilot/web/web.py", line 197, in index
    return render_template('index.html', async_mode=socketio.async_mode, pypilot_web_port=pypilot_web_port, tinypilot=tinypilot.tinypilot, translations=translations, language=config['language'], languages=Markup(LANGUAGES))
NameError: name 'LANGUAGES' is not defined
[2023-04-12 11:13:05,080] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/pypilot/web/web.py", line 197, in index
    return render_template('index.html', async_mode=socketio.async_mode, pypilot_web_port=pypilot_web_port, tinypilot=tinypilot.tinypilot, translations=translations, language=config['language'], languages=Markup(LANGUAGES))
NameError: name 'LANGUAGES' is not defined

I think it is when you removed the Greek translations

[seandepagnier]

The issue with gps is, some gps use binary mode and gpsd must read from them rather than pypilot. So in this case, it is good to prevent pypilot from probing those devices, and to give hints to gpsd to use them. Unfortunately it seems you have a device which was known as a potential gps, but in your case is actually a nmea0183 device. For this I am not sure what solution to deal with it as there is always going to have to be manual configuration to specify it (you have managed to do this but editing the udev rules is not supposed to be required)

As for updating, I fixed the issue with running pypilot.build twice I thought, but it seems that the way you have updated (a fresh clone) rather than pulling the existing repository into /home/tc/pypilot may be the reason.

I dont seem to get the error with LANGUAGES either which is not specific to removing greek but must be because of how you updated. I am not sure exactly why this occurred but for now I would suggest pulling instead of cloning.

[kagouraki]

Ok i did it. letting this info here for reference for what i did from fresh install of the tinypilot_beta_2024_01_14.img image:

cd
rm -f .ssh/knows_hosts
cd pypilot
git pull git@github.com:pypilot/pypilot.git
rm -rf web/translations/el
rm -rf hat/translations/el
python3 /hat/ugfx/generate_version.py 
. pypilot.build
sudo reboot

Thanks Sean for your help. You are awesome !

[seandepagnier]

the generate_version.py script is only used to generate the initial splash screen. You would also have to build with "make" and copy the splash to /opt It doesnt enhance the pilot in any way, but perhaps these steps should be integrated into pypilot.build

[kagouraki]

Yes i know what the generate_version.py script does. I read it. I wanted the true version to be displayed on splash so i made the steps above and now the splash version says 0.45 which is the updated one and the one reported in the pypilot info page. I didnt copy anything. I didnt run make. I just run pypilot.build.

[seandepagnier]

That is because I already did the steps to make that image.