UnicodeDecodeError still happens sometimes

[lsi8]

I am using latest version of pyvisa with Rigol DS1104Z oscilloscope and Raspberry Pi 4. I have omitted the last few lines of the following code (which would generate a plot) because the error is being generated by the last line only when I turn my sensing circuit on, i.e., only when the scope is exposed to a large sine wave signal:

import numpy
import matplotlib.pyplot as plot
import sys
import time
import pyvisa as visa

#Get the USB device, e.g. 'USB0::0x1AB1::0x0588::DS1ED141904883'
resources = visa.ResourceManager('@py')
usbDevices = list(filter(lambda x: 'USB' in x, resources.list_resources()))
if len(usbDevices) == 0:
    print("no usb devices found")
    sys.exit(-1)
print(usbDevices[0])
scope = resources.open_resource(usbDevices[0])
print(scope.query('*IDN?')) #Return the Rigol’s ID string to tell us it’s there

scope.timeout = 120
scope.write("*RST")
time.sleep(8)

scope.write(":CHAN2:DISP ON")
time.sleep(1)
scope.write(":CHAN3:DISP ON")
time.sleep(1)
scope.write(":CHAN1:PROB 1")
scope.write(":CHAN2:PROB 1")
scope.write(":CHAN3:PROB 1")
scope.write(":CHAN1:SCAL 0.1")
scope.write(":CHAN2:SCAL 0.002")
scope.write(":CHAN3:SCAL 0.050")
scope.write(":TIM:MAIN:SCAL 0.0002")
time.sleep(2)

# Get the timescale
timescale = float(scope.query(":TIM:SCAL?"))
print(timescale)

# Get the timescale offset
timeoffset = float(scope.query(":TIM:OFFS?"))
voltscale = float(scope.query(':CHAN1:SCAL?'))

# And the voltage offset
voltoffset = float(scope.query(":CHAN1:OFFS?"))

scope.write(":WAV:POIN:MODE RAW")
scope.write("WAV:DATA? CHAN1")
rawdata = scope.read_raw()[:int(1e7)]
data_size = len(rawdata)
sample_rate = scope.query(':ACQ:SRAT?')
print('Data size:', data_size, "Sample rate:", sample_rate)

# Start data acquisition again, and put the scope back in local mode:
scope.write(":RUN")
scope.write(":KEY:FORCE")
scope.close()

data = numpy.frombuffer(rawdata, 'B')

Here is the error message

USB0::6833::1230::DS1ZD223400795::0::INSTR
/usr/local/lib/python3.7/dist-packages/pyvisa_py/protocols/usbtmc.py:116: UserWarning: Unexpected MsgID format. Consider updating the device's firmware. See https://github.com/pyvisa/pyvisa-py/issues/20
  "Unexpected MsgID format. Consider updating the device's firmware. See https://github.com/pyvisa/pyvisa-py/issues/20"
RIGOL TECHNOLOGIES,DS1104Z Plus,DS1ZD223400795,00.04.04.SP4

0.0002
Traceback (most recent call last):
  File "/home/pi/Documents/projects/cu/pnaci/rigol/fuck.py", line 59, in <module>
    sample_rate = scope.query(':ACQ:SRAT?')
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 644, in query
    return self.read()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 486, in read
    message = self._read_raw().decode(enco)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xad in position 0: ordinal not in range(128)

Output of pyvisa-info

Machine Details:
   Platform ID:    Linux-5.4.51-v7l+-armv7l-with-debian-10.4
   Processor:      

Python:
   Implementation: CPython
   Executable:     /usr/bin/python3
   Version:        3.7.3
   Compiler:       GCC 8.3.0
   Bits:           32bit
   Build:          Jul 25 2020 13:03:44 (#default)
   Unicode:        UCS4

PyVISA Version: 1.11.3

Backends:
   ivi:
      Version: 1.11.3 (bundled with PyVISA)
      Binary library: Not found
   py:
      Version: 0.5.2
      ASRL INSTR: Available via PySerial (3.4)
      USB INSTR: Available via PyUSB (1.1.1). Backend: libusb1
      USB RAW: Available via PyUSB (1.1.1). Backend: libusb1
      TCPIP INSTR: Available 
      TCPIP SOCKET: Available 
      GPIB INSTR:
         Please install linux-gpib (Linux) or gpib-ctypes (Windows, Linux) to use this resource type. Note that installing gpib-ctypes will give you access to a broader range of funcionality.
         No module named 'gpib'

[MatthieuDartiailh]

The way you read the binary data is dangerous. read_raw will break on any termchar and will leave binary data in the buffer which is probably the cause of the next error. I suggest you try to use use the read_binary_values function or take inspiration from it if you do not want/cannot use this functio..

[lsi8]

Is there an up-to-date example showing how to get waveform data from an oscilloscope?

[MatthieuDartiailh]

That depends of the model of your oscilloscope. The manual for your model (https://beyondmeasure.rigoltech.com/acton/attachment/1579/f-0386/1/-/-/-/-/DS1000Z_Programming%20Guide_EN.pdf) describe the format starting line 240 and include what parameters enter into the conversion from the data returned to actual voltage. From there if you choose to use BYTE you need to use the datatype "b" with read_binary_values or "h" if you choose to use WORD. The documentation I linked in the pyvisa thread, describe how to get all the other parameters right for the transfer.

Let me know if you encounter any further issue.

[lsi8]

I haven't been able to get read_binary_values(datatype='b') to work. It produces the following error:

Traceback (most recent call last):
  File "/home/pi/Documents/projects/cu/pnaci/rigol/test.py", line 56, in <module>
    rawdata = scope.read_binary_values(datatype='b')
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 575, in read_binary_values
    block = self._read_raw(chunk_size)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 442, in _read_raw
    chunk, status = self.visalib.read(self.session, size)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/highlevel.py", line 519, in read
    return data, self.handle_return_value(session, status_code)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 251, in handle_return_value
    raise errors.VisaIOError(rv)
pyvisa.errors.VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before operation completed.

I have been able to get scope.query_binary_values(":WAV:DATA?", datatype='B') to work but only up to 400ish datapoints... Anything over 500 datapoints and I get the following error:

Traceback (most recent call last):
  File "/home/pi/Documents/projects/cu/pnaci/rigol/test.py", line 56, in <module>
    rawdata = scope.query_binary_values("WAV:DATA?", datatype='B')
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 754, in query_binary_values
    chunk_size,
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 602, in read_binary_values
    self.read_bytes(expected_length - len(block), chunk_size=chunk_size)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/resources/messagebased.py", line 371, in read_bytes
    chunk, status = self.visalib.read(self.session, size)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/highlevel.py", line 512, in read
    data, status_code = self.sessions[session].read(count)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/usb.py", line 156, in read
    USBTimeoutException,
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/sessions.py", line 793, in _read
    current = reader()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/usb.py", line 133, in _usb_reader
    return self.interface.read(count)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/protocols/usbtmc.py", line 465, in read
    response = BulkInMessage.from_bytes(resp)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa_py/protocols/usbtmc.py", line 113, in from_bytes
    msgid, btag, btaginverse = struct.unpack_from("BBBx", data)
struct.error: unpack_from requires a buffer of at least 4 bytes

[MatthieuDartiailh]

I am not sure how why going from signed to unsigned changes the number of point. However the second error looks like either an issue in pyvisa-py or another oddity from Rigol ? As I suggested in your pyvisa-py issue do you have access to a machine where you can install Keysight or NI Visa ?

[lsi8]

I will install NI Visa on my mac and attempt to save waveform data today.

[lsi8]

Could the problem be chunk_size?