[COM] Communication issue with Siglent SDS1104X-E using USB

[WillMatthews]

Instrument details

• Model: Siglent SDS 1104X-E
• Communication: USB (usbtmc I if I am sane?)
• Link to the documentation: https://siglentna.com/wp-content/uploads/dlm_uploads/2017/10/ProgrammingGuide_forSDS-1-1.pdf

Output of pyvisa-info

Machine Details:
   Platform ID:    Linux-5.4.0-99-generic-x86_64-with-Ubuntu-18.04-bionic
   Processor:      x86_64

Python:
   Implementation: CPython
   Executable:     /usr/bin/python3
   Version:        3.6.9
   Compiler:       GCC 8.4.0
   Bits:           64bit
   Build:          Dec  8 2021 21:08:43 (#default)
   Unicode:        UCS4

PyVISA Version: 1.11.3

Backends:
   ivi:
      Version: 1.11.3 (bundled with PyVISA)
      Binary library: Not found
   py:
      Version: 0.5.2
      ASRL INSTR: Available via PySerial (3.5)
      USB INSTR: Available via PyUSB (1.1.1). Backend: libusb1
      USB RAW: Available via PyUSB (1.1.1). Backend: libusb1
      TCPIP INSTR: Available 
      TCPIP SOCKET: Available 
      GPIB INSTR:
         Please install linux-gpib (Linux) or gpib-ctypes (Windows, Linux) to use this resource type. Note that installing gpib-ctypes will give you access to a broader range of funcionality.
         No module named 'gpib'

The problem...

I got this traceback, which I've seen mentioned around with no good fix for..

  File "./test-visa.py", line 172, in <module>
    data = scope.query_binary_values("C1:WF? DAT2", datatype='b')
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa/resources/messagebased.py", line 754, in query_binary_values
    chunk_size,
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa/resources/messagebased.py", line 602, in read_binary_values
    self.read_bytes(expected_length - len(block), chunk_size=chunk_size)
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa/resources/messagebased.py", line 371, in read_bytes
    chunk, status = self.visalib.read(self.session, size)
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/highlevel.py", line 512, in read
    data, status_code = self.sessions[session].read(count)
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 156, in read
    USBTimeoutException,
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/sessions.py", line 793, in _read
    current = reader()
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 133, in _usb_reader
    return self.interface.read(count)
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 466, in read
    response = BulkInMessage.from_bytes(resp)
  File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 114, in from_bytes
    msgid, btag, btaginverse = struct.unpack_from("BBBx", data)
struct.error: unpack_from requires a buffer of at least 4 bytes

I've been dashing my brain out over rocks on this problem for quite a while now, I contacted Siglent themselves and they couldn't offer any help.

I appear to get approx. 60k points from the oscilloscope, a little shorter than the expected 14M points the scope claims to have in its memory. I found this by putting a print statement in from_bytes (usbtmc.py) and analysing the output. the last line seems to cause the exception, as it's empty: b''

On my waveform query, the output format appears like it doesn't match the packing specified in from_bytes's struct.unpack_from. The struct.unpack_from in from_bytes (usbtmc.py) has a packing of "BBBx", however I get far more, valid, points if I use the packing "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" which seems weird to me - I'm even sure if I'm doing this correctly or if this provides anything insightful, as I don't have any other instruments to compare to.

Here are a couple lines I've captured from somewhere in the middle of the waveform response.

b'\xf3\xec\xe7\xe4\xe3\xe4\xe6\xe8\xea\xec\xed\xed\xed\xed\xee\xf0\xf2\xf6\xfb\x01\x06\t\x0b\n\x06\x00\xf7\xee\xe5\xdc\xd6\xd3\xd3\xd5\xda\xe1\xe8\xf0\xf7\xfe\x04\x08\x0c\x0e\x10\x12\x13\x14\x14\x14\x14\x13\x11\x10\x0e\x0c\n\x08\x06\x04\x03\x01\xfe\xfb'
b'\xf8\xf3\xef\xeb\xe7\xe5\xe4\xe4\xe6\xea\xee\xf2\xf6\xfa\xfd\x00\x02\x03\x03\x04\x04\x04\x04\x04\x04\x03\x01\xff\xfd\xfb\xf9\xf7\xf7\xf8\xfb\xff\x04\t\x0e\x13\x16\x18\x18\x16\x12\r\x08\x03\xff\xfc\xfb\xfc\xfe\x01\x05\x08\x0b\x0c\x0c\n\x07\x02\xfc\xf6'
b'\xf1\xec\xe9\xe8\xe9\xeb\xf0\xf5\xfa\x01\x06\x0b\x0f\x11\x12\x12\x11\x11\x11\x12\x14\x17\x1c!%*-//-)$\x1d\x16\x0f\x08\x02\xfe\xfb\xfa\xfa\xfa\xfb\xfd\xfe\xff\x00\x00\xff\xfe\xfd\xfc\xf9\xf7\xf4\xf2\xef\xed\xeb\xe9\xe9\xe9\xea\xec'
b'\xef\xf2\xf6\xfa\xfe\x02\x04\x06\x07\x08\x08\x08\x08\x08\t\x0b\r\x0f\x11\x13\x14\x14\x13\x12\x0f\x0c\t\x06\x03\x01\xff\xff\xfe\xff\xff\x00\x00\x00\x00\xff\xff\xff\xff\x01\x03\x05\x08\x0c\x0f\x11\x13\x13\x12\x0f\x0b\x06\x02\xfd\xf9\xf7\xf5\xf5\xf6\xf9'
b'\xfc\x00\x04\x06\x08\x08\x07\x05\x01\xfd\xf9\xf5\xf2\xef\xed\xec\xeb\xeb\xeb\xeb\xea\xea\xeb\xec\xee\xf1\xf5\xfa\x00\x07\r\x12\x16\x18\x18\x16\x12\x0c\x06\xff\xfa\xf5\xf3\xf3\xf6\xfa\x00\x07\x0e\x14\x18\x1a\x1a\x17\x13\x0c\x05\xfe\xf8\xf3\xef\xee\xee\xef'
b'\xf2\xf5\xf8\xfa\xfc\xfe\xfe\xfd\xfc\xfb\xf9\xf8\xf6\xf4\xf3\xf2\xf1\xf0\xf0\xf0\xf0\xf1\xf2\xf4\xf6\xf7\xf9\xfb\xfb\xfb\xfa\xf8\xf6\xf3\xf0\xee\xed\xed\xee\xf0\xf4\xf8\xfc\x00\x04\x06\x07\x06\x04\x01\xfd\xf8\xf4\xef\xea\xe7\xe3\xe1\xe0\xe0\xe0\xe1\xe3\xe6'
b'\xe9\xed\xf1\xf5\xf9\xfe\x02\x05\x08\n\x0b\x0c\r\r\r\r\x0c\x0c\x0c\x0c\x0b\x0b\x0b\n\x08\x06\x03\x00\xfc\xf7\xf3\xee\xea\xe6\xe4\xe3\xe2\xe3\xe6\xea\xf0\xf6\xfd\x04\n\x10\x14\x16\x15\x12\x0b\x03\xfa\xf1\xe9\xe3\xdf\xdf\xe2\xe9\xf2\xfc\x06\x10'
b'\x17\x1b\x1c\x1a\x15\x0e\x06\xfe\xf7\xf2\xef\xee\xee\xef\xf0\xf0\xf0\xee\xec\xea\xe7\xe6\xe5\xe6\xe8\xec\xf0\xf5\xf8\xfb\xfc\xfb\xf9\xf6\xf2\xee\xeb\xea\xe9\xea\xec\xee\xf1\xf3\xf5\xf6\xf7\xf7\xf7\xf7\xf7\xf8\xf9\xfc\xfe\x01\x04\x06\x08\n\x0b\x0b\n\t'
b'\x08\x07\x05\x05\x04\x05\x06\t\x0c\x0f\x13\x17\x1b\x1d\x1f\x1f\x1e\x1b\x16\x10\t\x03\xfc\xf7\xf3\xf1\xf1\xf2\xf4\xf7\xf9\xfc\xfe\xff\xff\xff\xfe\xfd\xfc\xfc\xfd\xff\x01\x04\x08\x0b\x0e\x11\x12\x12\x12\x11\x0f\x0e\x0b\t\x08\x07\x06\x06\x07\x07\t\x0b'

I'm not really sure what else I can do - I think I've reached the extent I can just poking around by myself, please may I have some help? Many Thanks in advance, William

[MatthieuDartiailh]

That's a tough one. I know some manufacturers implement USBTMC loosely, which is why the BulkInMessage has a from quirky method, but apparently it does not help in your case.

Since you are running Ubuntu you can try to install the VISA distribution of Keysight and see if things work with the ici backend. If it does it would be great to see a Wireshark comparison between the two backends to figure out what pyvisa-py is doing wrong.

[WillMatthews]

Thank you! I know some USBTMC implementations can be a bit janky. What you suggest is a great idea. Before I implement that, I managed to get a wireshark capture of serial communications through pyvisa. I will update soon once I have the Keysight VISA installed and working.

pyvisapy_usbtmc_wireshark.zip

[WillMatthews]

Made some progress!

I had a poke around in ./pyvisa_py/protocols/usbtmc.py On line 440 is the definition of the method read(self,size) In this method's main loop iterating through requests and parsing the data, beginning at approx line 457, sometimes resp can have zero length.

I added a check as follows in the loop just after receiving data and before it's parsed with from_bytes:

if len(resp) == 0:
  continue

and I get far more data than before, but it eventually ends up in a write timeout (sometimes?). Now if I use read_bytes(1000000) it's able to get 1M points, however when attempting to get the full 14M points I also end up in a write timeout.

I'll have a go with the Keysight VISA and grab a wireshark capture soon to see what they do differently. I'm spread a bit thin at the moment (writing up thesis) so regrettably this isn't my priority number 1. I'll get around to exploring more when I have time.

[MatthieuDartiailh]

Thanks for your investigation.

Could you post the timeout traceback and check if adding a 1ms sleep before the continue helps in case we are saturating the instrument somehow.

[WillMatthews]

I found that once this timeout occurs, the o'scope must be restarted in order to communicate with it successfully again. It looks like data points are still in the scope's memory - but when running a query it just continues providing the point data.

Timeout Traceback (Click To Expand)

``` Traceback (most recent call last): File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 467, in read resp = raw_read(recv_chunk + header_size + max_padding) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 271, in read data = self.usb_recv_ep.read(size, self.timeout).tobytes() File "/home/will/.local/lib/python3.6/site-packages/usb/core.py", line 423, in read return self.device.read(self, size_or_buffer, timeout) File "/home/will/.local/lib/python3.6/site-packages/usb/core.py", line 1024, in read self.__get_timeout(timeout)) File "/home/will/.local/lib/python3.6/site-packages/usb/backend/libusb1.py", line 851, in bulk_read timeout) File "/home/will/.local/lib/python3.6/site-packages/usb/backend/libusb1.py", line 954, in __read _check(retval) File "/home/will/.local/lib/python3.6/site-packages/usb/backend/libusb1.py", line 602, in _check raise USBTimeoutError(_strerror(ret), ret, _libusb_errno[ret]) usb.core.USBTimeoutError: [Errno 110] Operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test-pyvisa.py", line 72, in data = scope.read_bytes(100000) File "/home/will/.local/lib/python3.6/site-packages/pyvisa/resources/messagebased.py", line 371, in read_bytes chunk, status = self.visalib.read(self.session, size) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/highlevel.py", line 512, in read data, status_code = self.sessions[session].read(count) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 156, in read USBTimeoutException, File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/sessions.py", line 793, in _read current = reader() File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 133, in _usb_reader return self.interface.read(count) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 478, in read self._abort_bulk_in(self._btag) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 381, in _abort_bulk_in timeout=abort_timeout_ms, File "/home/will/.local/lib/python3.6/site-packages/usb/core.py", line 1079, in ctrl_transfer self.__get_timeout(timeout)) File "/home/will/.local/lib/python3.6/site-packages/usb/backend/libusb1.py", line 901, in ctrl_transfer timeout)) File "/home/will/.local/lib/python3.6/site-packages/usb/backend/libusb1.py", line 604, in _check raise USBError(_strerror(ret), ret, _libusb_errno[ret]) usb.core.USBError: [Errno 32] Pipe error ```

[WillMatthews]

It looks like the 1ms pause fixes this! I will check more tonight and hopefully get a pull request in soon. Many many thanks Matthieu :)

[WillMatthews]

Spoke too soon! I still have the timeout traceback as above when doing read_binary_values()

I also had this traceback when doing read_bytes(14000000), which I haven't yet seen I think...

Click to expand Traceback

``` Traceback (most recent call last): File "test-pyvisa.py", line 72, in data = scope.read_bytes(1400000) File "/home/will/.local/lib/python3.6/site-packages/pyvisa/resources/messagebased.py", line 371, in read_bytes chunk, status = self.visalib.read(self.session, size) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/highlevel.py", line 512, in read data, status_code = self.sessions[session].read(count) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 156, in read USBTimeoutException, File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/sessions.py", line 793, in _read current = reader() File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/usb.py", line 133, in _usb_reader return self.interface.read(count) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 473, in read response = BulkInMessage.from_bytes(resp) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 118, in from_bytes return BulkInMessage.from_quirky(data) File "/home/will/.local/lib/python3.6/site-packages/pyvisa_py/protocols/usbtmc.py", line 133, in from_quirky transfer_size, transfer_attributes = struct.unpack_from("

[MatthieuDartiailh]

You may have to add the smae logic to from_quirky since often in long transfer the USB is weirdly formatted and we cannot go through from_bytes. Having the wait in both functions may help.

[WillMatthews]

Adding a sleep in from_quirky didn't seem to help. What (maybe?) helped was calling read_bytes(128) over and over. Not sure what to make of this yet but I will look more in detail.

[WillMatthews]

looking at the documentation on page 163 on the vi.read example they have, they use a maximum count (MAX_CNT) of 200. When I try chunking with read_bytes(200) it seems to work better than read_bytes(128) or read_bytes(64). I will have to update on this later, as I'm still trying to see if this workaround actually works. This method is extremely slow, however.