Cannot connect to device

[fuzzy-focus]

I have the device connected, it is recognized, and I can interface with it correctly over screen.

When I use this package the device is not discovered. When I try to connect to it directly I get this:

Traceback (most recent call last):
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 327, in _query
    self._read_string(expected_echo)
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 352, in _read_string
    raise NumatoGpioError(string)
numato_gpio.NumatoGpioError: d t
000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 116, in __init__
    _ = self.id
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 138, in id
    self._id = self._read_int32("id get")
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 356, in _read_int32
    self._query(query)
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 329, in _query
    raise NumatoGpioError(
numato_gpio.NumatoGpioError: Query 'id get' returned unexpected echo d t
000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "gpio-test.py", line 6, in <module>
    dev = gpio.NumatoUsbGpio('/dev/ttyACM0')
  File "/home/bud8rt/.local/share/virtualenvs/brc3-test-automation-Cum7zfbO/lib/python3.8/site-packages/numato_gpio/__init__.py", line 121, in __init__
    raise NumatoGpioError(
numato_gpio.NumatoGpioError: Device /dev/ttyACM0 doesn't answer like a numato device: Query 'id get' returned unexpected echo d t
000

I am using version 0.9.0-dev and the 8Port Numato GPIO USB Port.

[clssn]

Thanks for reporting this. Since I have only 32 port numato devices, are you willing to assist in finding the issue? Since there's no emulator I have no way to test this myself.

[fuzzy-focus]

I'll gladly assist! I assume the responses differ somehow. What can I do?

[clssn]

Okay, with screen, what does id get and ver return?

[fuzzy-focus]

I get this

sudo screen /dev/ttyACM0

>id get
00000000
>ver
00000008

[clssn]

Thanks, looks good so far. I found some time and created a troubleshooting document and script (on master already). I'd need the output of the troubleshooting script for further investigation.

[clssn]

By the way, I rebased the support-all-numato-usb-gpio-devices branch, so make sure to git reset --hard origin/support-all-numato-usb-gpio-devices to get the script.

[fuzzy-focus]

I checkout out the current origin/support-all-numato-usb-gpio-devices branch, disconnected and reconnected the device, gave it access rights, and ran the troubleshoot script.

(numato-gpio) ➜  doc git:(support-all-numato-usb-gpio-devices) ✗ sudo chmod 777 /dev/ttyACM0                    
[sudo] password for bud8rt: 
(numato-gpio) ➜  doc git:(support-all-numato-usb-gpio-devices) ✗ python -m numato_gpio.troubleshoot /dev/ttyACM0
Testing device /dev/ttyACM0 with pyserial==3.5
Traceback (most recent call last):
  File "/home/bud8rt/.local/share/virtualenvs/numato-gpio-C7HQuqfx/lib/python3.8/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
OSError: [Errno 16] Device or resource busy: '/dev/ttyACM0'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bud8rt/workspace/numato-gpio/src/numato_gpio/troubleshoot.py", line 43, in <module>
    main()
  File "/home/bud8rt/workspace/numato-gpio/src/numato_gpio/troubleshoot.py", line 33, in main
    ser = serial.Serial(device, 19200, timeout=0.1)
  File "/home/bud8rt/.local/share/virtualenvs/numato-gpio-C7HQuqfx/lib/python3.8/site-packages/serial/serialutil.py", line 244, in __init__
    self.open()
  File "/home/bud8rt/.local/share/virtualenvs/numato-gpio-C7HQuqfx/lib/python3.8/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 16] could not open port /dev/ttyACM0: [Errno 16] Device or resource busy: '/dev/ttyACM0'

The Device or resource busy error is new and only appeared after checking out the new branch. I did not interact with the device at all before running the script.

[clssn]

You can find out which process opened the device with lsof | grep /dev/ttyACM0 (list open files). In the first column you'll find the name of the process. Afaik modemmanager might open ttyACM devices because they are often modems and try to discover them with those AT commands. Maybe that's the case for you. But maybe you didn't cleanly shut your last screen session on the device... ;-) I'm curious!

By the way, I wouldn't set the executable bits on the device. So 666 should be enough. The cleaner way would be sudo adduser $USER dialout if the device belongs to the dialout group. So you don't have to change access permissions after each reconnect.

[fuzzy-focus]

Fair enough, entirely my bad, screen was still open. Thought I had kill -9d that thing, but alas, no!

so now the result is this:

python -m numato_gpio.troubleshoot /dev/ttyACM0
Testing device /dev/ttyACM0 with pyserial==3.5
writing to /dev/ttyACM0: b'gpio notify off\r'
response (18 byte):
b'gpio notify off\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x6e 0x6f 0x74 0x69 0x66 0x79 0x20 0x6f 0x66 0x66 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'id get\r'
response (19 byte):
b'id get\n\r00000000\n\r>'
0x69 0x64 0x20 0x67 0x65 0x74 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'ver\r'
response (16 byte):
b'ver\n\r00000008\n\r>'
0x76 0x65 0x72 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x38 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask ffffffff\r'
response (20 byte):
b'gpio iomask ffffffff'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x66 0x66 0x66 0x66 0x66 0x66 0x66 0x66
--
writing to /dev/ttyACM0: b'gpio iomask 00000000\r'
response (5 byte):
b' pio '
0x20 0x70 0x69 0x6f 0x20
--
writing to /dev/ttyACM0: b'gpio readall\r'
response (3 byte):
b'\n\r>'
0xa 0xd 0x3e
--
(numato-gpio) ➜  doc git:(support-all-numato-usb-gpio-devices) ✗ python -m numato_gpio                          
Discovered devices: (None)

I am now in the dialout group and that is a lot cleaner. Thanks for the tipp. Was mostly just me going "freakin let me do what I want", without much consideration behind it :sweat_smile:

Btw., not sure if it is because it doesn't work for me, but the poll thread does not usually shut down cleanly. Have you considered starting it as a daemon thread?

[clssn]

Thanks for the hint regarding the daemon thread. This option was not on my radar. I'll give it a try soon. Regarding your device's response: 1) The 8 port device doesn't support the notify mode (that was new to me), therefore it doesn't confirm to switch notifications off. I hope the sheer attempt doesn't render the device inconsistent. 2) You've got a firmware version 8, while I have 9. Though I guess, the firmwares for different devices are independently versioned, so this shouldn't mean anything 3) My troubleshoot script didn't consider the port width. I fixed that, adding a ports argument to the troubleshooting script. Please re-run it like python3 -m numato_gpio.troubleshoot /dev/ttyACM0 8

[clssn]

Made the poll thread a daemon thread. Now you shouldn't experience the related problems anymore do you?

[fuzzy-focus]

(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio.troubleshoot /dev/ttyACM0 8
Testing device /dev/ttyACM0 with pyserial==3.5

writing to /dev/ttyACM0: b'id get\r'
response (19 byte):
b'id get\n\r00000000\n\r>'
0x69 0x64 0x20 0x67 0x65 0x74 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'ver\r'
response (16 byte):
b'ver\n\r00000008\n\r>'
0x76 0x65 0x72 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x38 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask ff\r'
response (17 byte):
b'gpio iomask ff\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x66 0x66 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask 00\r'
response (17 byte):
b'gpio iomask 00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio readall\r'
response (19 byte):
b'gpio readall\n\r00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x72 0x65 0x61 0x64 0x61 0x6c 0x6c 0xa 0xd 0x30 0x30 0xa 0xd 0x3e
--
(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio
Discovered devices: (None)

The daemon thread worked nicely.

[clssn]

Your device responds with the same strange eol sequence \n\r as the reporter of issue #9. So I rebased the branch you're using onto the fix-branch for that similar issue. So please carefully check out the rebased version of this branch (e.g. resetting your local branch to the origin's version) and re-run the test.

[fuzzy-focus]

~/dev/numato-gpio >>> git fetch && git reset --hard origin/support-all-numato-usb-gpio-devices
HEAD is now at 12045d9 Make poll thread a daemon

(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio.troubleshoot /dev/ttyACM0 8
Testing device /dev/ttyACM0 with pyserial==3.5

writing to /dev/ttyACM0: b'id get\r'
response (19 byte):
b'id get\n\r00000000\n\r>'
0x69 0x64 0x20 0x67 0x65 0x74 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'ver\r'
response (16 byte):
b'ver\n\r00000008\n\r>'
0x76 0x65 0x72 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x38 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask ff\r'
response (17 byte):
b'gpio iomask ff\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x66 0x66 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask 00\r'
response (17 byte):
b'gpio iomask 00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio readall\r'
response (19 byte):
b'gpio readall\n\r00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x72 0x65 0x61 0x64 0x61 0x6c 0x6c 0xa 0xd 0x30 0x30 0xa 0xd 0x3e
--
(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio
Discovered devices: (None)

So sadly that still didn't work.

[clssn]

I certainly was too tired to push my changes. Therefore you tested the same thing as before. Sorry for that. I'll push in a minute.

[fuzzy-focus]

Well glad I'm not the only one stuff like that happens to....

got the changes again, now it fails differently! It just waits indefinitely, trying to acquire a lock. Had to CTRL+C it.

(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio.troubleshoot /dev/ttyACM0 8
Testing device /dev/ttyACM0 with pyserial==3.5

writing to /dev/ttyACM0: b'id get\r'
response (19 byte):
b'id get\n\r00000000\n\r>'
0x69 0x64 0x20 0x67 0x65 0x74 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'ver\r'
response (16 byte):
b'ver\n\r00000008\n\r>'
0x76 0x65 0x72 0xa 0xd 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x38 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask ff\r'
response (17 byte):
b'gpio iomask ff\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x66 0x66 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio iomask 00\r'
response (17 byte):
b'gpio iomask 00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x69 0x6f 0x6d 0x61 0x73 0x6b 0x20 0x30 0x30 0xa 0xd 0x3e
--
writing to /dev/ttyACM0: b'gpio readall\r'
response (19 byte):
b'gpio readall\n\r00\n\r>'
0x67 0x70 0x69 0x6f 0x20 0x72 0x65 0x61 0x64 0x61 0x6c 0x6c 0xa 0xd 0x30 0x30 0xa 0xd 0x3e
--
(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio
^CTraceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/db/dev/numato-gpio/src/numato_gpio/__main__.py", line 17, in <module>
    main()
  File "/home/db/dev/numato-gpio/src/numato_gpio/__main__.py", line 8, in main
    discover()
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 62, in discover
    gpio = NumatoUsbGpio(device_file)
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 121, in __init__
    self.notify = False
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 263, in notify
    self._query(query, expected=expected_response)
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 364, in _query
    self._read_string(expected)
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 379, in _read_string
    string = self._read(len(expected.encode()))
  File "/home/db/dev/numato-gpio/src/numato_gpio/__init__.py", line 410, in _read
    self._can_read.wait()
  File "/usr/lib/python3.9/threading.py", line 312, in wait
    waiter.acquire()
KeyboardInterrupt

[clssn]

I extended the unit tests to reproduce the above error. Since your device doesn't have the notification feature, it doesn't respond to 'gpio notify' commands. That's where you were stuck above. Waiting for the notification status. Fixed that. Please re-test - and don't be angry if you uncover the next problem, please ;-)

[fuzzy-focus]

I wasn't angry at all, I'm happy that we are on the journey of getting this to work!

In that context:

HEAD is now at 5cbe9a1 Handle missing notify capability in 8 port devices
(numato-gpio) ~/dev/numato-gpio >>> python -m numato_gpio
Discovered devices: 
dev: /dev/ttyACM0 | id: 0 | ver: 8 | ports: 8 | iodir: 0xff | iomask: 0x00 | state: 0x00

Success!

I will do some more extensive testing later, but for now, this is great.

I assume the changes required for my device are incompatible with the normal package and can't really be merged?

[clssn]

Great! I'm really happy we did this together! But your assumption is wrong. I implemented this in a generic way that should be compatible with all devices from 8 to 128 bit. After some more testing (maybe extend the device mockup to support notifications testing) I'll definitely merge this branch back to master and release it. Not today, but soon (because it's not gonna be easier the longer I wait...).

Please tell me about your results while testing. I'm interested in the smallest issue. I'll leave this ticket open for now.

[clssn]

Froma tested the branch with his 8, 16 and 32 port devices and they worked. Im quite positive now that the other devices work as well and will probably release the branch soon. Let me know if you find anything to fix.

[fuzzy-focus]

Hey, sorry that took me quite a while, had to resolder the circuit I use with the GPIO port.

It works perfectly from what I could test (did only use the ports as outputs)!

So yeah, go for the release :)

[clssn]

Hi, writing some tests for notifications, I got aware of some corner cases I couldn't cover correctly. So release will take until I fixed this. But be sure, I'm on it!

[clssn]

Hi @fuzzy-focus, I fixed the notifications (got unit-tests for them now). Before I release this, I'd love you to re-test whether everything still works with your device (indicating, that my unit-tests cover the '\n\r' variant correctly).

[fuzzy-focus]

I can obviously not test the full capability, but discovery, connection, setting modes and pushing buttons works as desired and without issues or warnings!

I tested commit #036610f

[clssn]

Thanks @fuzzy-focus ! I'll release 0.10.0 very soon!

[clssn]

Closing the issue. Please raise a new one should you be in trouble again!