Memory overflows when working with several joulescopes

[imbuedhope]

I have a pretty straight forward script that uses Device.statistics_callback_register() to push results from every joulescope connected to the computer to a standard queue.Queue which I loop over to write to a database in the main thread. Its setup very similar to the example scripts.

The script runs well in general, but when I have 4 scopes connected to the computer the underlying thread seems to block for so long that the queue fills up faster than it gets cleared (the function calls to process each entry in the queue takes about .005 seconds each on their own). This eventually results in the queue growing to take up all the memory in the system. The experimental setup I'm working with sets a max of 12 scopes active at once so I'm a bit stumped.

I currently have sampling_frequency = 'auto', buffer_duration = 15 (since it ate a lot of ram at 30), and reduction_frequency = 2 in an attempt to mimic data coming in at 1 Hz from 8 scopes as a benchmark.

I'm willing to reduce the reduction_frequency by a large enough margin to get this to work, but I can't seem to find a way to get it lower than 1 Hz. Setting the sampling_frequency to anything other than auto seems to trigger a slew process: stream_buffer is behind messages.

Any recommended solutions?

Currently considering / trying out using multiprocessing to have each device managed in its own process, but reducing the rate of the data entering the queue would be a lot cleaner.

[mliberty1]

Hi @imbuedHope

When streaming full-rate data with more than 2 Joulescopes connected to a computer, you have to worry about USB bandwidth. 3 Joulescopes to the same USB root hub usually work, but most root hubs start struggling with 4 (32 MB/s of 53 MB/s theoretical max). The other issue is the Python GIL, which prevents threads from running simultaneously. As you mentioned, multiprocessing is one way around the GIL.

However, it sounds like you only need statistics, not full-rate data. You can use the "on-instrument" statistics computation. These on-instrument statistics are fixed at 2 Hz and do not include variance (standard deviation). Other than that, they are the same as the host-side computed statistics. You select the source of the statistics when your code registers, like this:

device.statistics_callback_register(cbk, 'sensor')

We have a few examples:

• python3 -m joulescope statistics
• statistics.py
• accumulators.py

Now, the other part is your thread that writes to the database. If each entry takes 0.005 seconds to process, then you have

0.005 seconds / entry * 2 entries / (second * device) * 12 devices 

= 0.12 (12%) of the total processing time

Which seems ok. If you need to further reduce the data rate, you can combine statistics for a single Joulescope instrument on either side of the queue. Since you don't have variance, the math is pretty easy:

v_mean = sum(v_means) / count
v_min = min(v_mins)
v_max = max(v_maxs)
energy & charge are accumulated, just use the most recent statistic

Does this make sense? Does it work for you?

[imbuedhope]

Thanks for tip. Switching to the sensor for statistics did the trick.