stream_notify() exception on Raspberry Pi

[ghost]

Hi !

I'm currently working on a script to monitor and store Joulescope data over a long period using a Raspberry Pi. I tested my script on Windows and it works without any issue, but on RPi, after a few minutes I get this exception:

<joulescope.jls_v2_writer.JlsWriter object at 0x7f93681f40> stream_notify() exception - stop streaming
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/joulescope/driver.py", line 792, in _on_process
    rv |= bool(obj.stream_notify(self.stream_buffer))
  File "/usr/local/lib/python3.9/dist-packages/joulescope/jls_v2_writer.py", line 151, in stream_notify
    self._wr.fsr_f32(idx, self._idx, x)
  File "pyjls\binding.pyx", line 299, in pyjls.binding.Writer.fsr_f32
  File "pyjls\binding.pyx", line 322, in pyjls.binding.Writer.fsr
RuntimeError: fsr failed 19

My script looks like this (irrelevant part omitted):

with device:

        device.statistics_accumulators_clear()
        device.statistics_callback_register(statistics_callback)

        while some_condition:

            filename = generate_filename()

            # Build JLS Writer
            with JlsWriter(device, filename) as p:

                # Register recorder
                device.stream_process_register(p)

                # Wait until it's time
                while not timing_condition:
                    time.sleep(0.01)

                print_statistics()

                # Unregister recorder
                device.stream_process_unregister(p)

I also tried using a DataRecorder instead of a JlsWriter, but I got a similar issue.

Here are more information about my environment:

$ python -VV
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]

$ joulescope info
System information
    Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
    Platform: Linux-5.15.61-v8+-aarch64-with-glibc2.31 (linux)
    Processor:
    executable: /usr/bin/python3
    frozen: False

joulescope version: 0.9.11
Found 1 connected Joulescope:
    Joulescope:002275   ctl=1.3.4            sensor=

$ pip show joulescope pyjls numpy
Name: joulescope
Version: 0.9.11
Summary: Joulescope™ host driver and utilities
Home-page: https://joulescope.readthedocs.io
Author: Jetperch LLC
Author-email: joulescope-dev@jetperch.com
License: Apache 2.0
Location: /home/ubiwan/.local/lib/python3.9/site-packages
Requires: psutil, python-dateutil, pyjls, pymonocypher, numpy
Required-by:
---
Name: pyjls
Version: 0.4.2
Summary: Joulescope™ file format
Home-page: https://joulescope.readthedocs.io
Author: Jetperch LLC
Author-email: joulescope-dev@jetperch.com
License: Apache 2.0
Location: /home/ubiwan/.local/lib/python3.9/site-packages
Requires: numpy
Required-by: joulescope
---
Name: numpy
Version: 1.23.4
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /usr/local/lib/python3.9/dist-packages
Requires:
Required-by: pyjls, joulescope

[mliberty1]

The error code is 19 from the jls library. It would be nice if this displayed more info than the error code, but looking at jls/ec.h shows that 19 is JLS_ERROR_BUSY. Looking at threaded_writer.c, this only happens when the file writing cannot keep up with the incoming data rate.

What storage media are you writing to? If it's an SD card, what is the speed rating? It needs to support at least 10 MB/s write, sustained.

[ghost]

Thanks for your quick reply !

I tried both on the Class 10 SD card where the OS is installed and on a USB key formatted in FAT32 plugged on one of the USB3 ports.

Maybe lowering the sampling rate would work ? I'll try it tomorrow.

[mliberty1]

I have not tried writing to SD cards. I also forgot that JLS v2 requires approximately 16 MB/s (60 GB/hr) at full rate. You do need to ensure you have a large enough storage device. Any storage device that can record 4k video should be able to handle full-rate Joulescope data.

Reducing the sampling rate should definitely help. Let me know what you find!

[ghost]

By measuring using dd, I get a 66MB/s writing speed on my USB key and it has a 64GB capacity.

For sampling frequencies >= 200kHz I get stream_buffer is behind warnings every seconds + consistently get the JLS_ERROR_BUSY error For sampling frequencies >= 100kHz I only get stream_buffer is behind warnings every seconds

For lower sampling requencies, I only get the stream_buffer is behind warning when the JlsWriter is being closed.

Maybe it is related to the Python interpreter IO performances rather than the file system ?

[mliberty1]

Well, the JLS v2 file format is all native, high-performance C code:

https://github.com/jetperch/jls

That said, the format is definitely optimized for read, not write. The format does require rewriting some headers, and we rely on the OS & filesystem to buffer. I am not familiar with optimizing USB flash drive performance under Linux, but you could possibly investigate RAM filesystem buffering options.

Flash drives are also weird in that they can have significant delays while reclaiming old storage. You could consider an external USB HDD or SSD.

[mliberty1]

May something here will help:

https://github.com/raspberrypi/linux/issues/1753

[ghost]

I will take a look thanks ! For my use case, I will probably just use a low sampling frequency (probably 20kHz) to reduce the amount of data stored anyway.

[mliberty1]

Sounds good! I will close this issue, but please feel free to open it if you want more assistance.