Recording Data Queue Overflow and data corruption

[JoseGuzman]

Hi,

I am running the repository version of the Open Ephys GUI installed from sources in a Ubuntu 18.04 with 24 cores and 128 G of memory. I use an Intan 512ch recording controlled connected to a USB3 port.

Starting the recording to save in binary 64 channels with a relatively simple Signal chain (Intan recording, Channel mapping, and LFP viewer, as seen here:

Just after a few minutes of recording (sometimes below 30 seg), I get a message that prompts Recording Data Queue Overflow. My data is then corrupted, and at some random times, zero values are introduced.

Also, if I record longer, I get the message below.

I even tested without any other USB port connected, and get the same error.

I am very worried about this issue and would strongly appreciate some help!.

Best

[Claybarn]

Hey Jose,

Upon initializing the Intan board, you should see some output on the console along the lines of:

"Expecting blocksize of ... for ... streams"

There is some useful information around there. Would help diagnose the problem if you provided that!

Best,

Clayton

[jsiegle]

A few more questions:

• Are you writing data to an SSD or a hard drive? A hard drive may not be fast enough to keep up with 512 channels
• Do you see the same problem if you remove the Channel Map and Bandpass Filter from the signal chain?

[JoseGuzman]

Thank you SO MUCH!!! for your fast response. I'm starting to have some things clear. The problem probably arrives from the hard drive!

• Having only the Intan Rec Controller alone, or (Int Rec Controller + Channel map) does not make the problem any after 20 min. The problem arises immediately (< 1min) when I add the bandpass filter, and I put the 67 channels to be recorded.
• I thought I had 2 hard disks HHD of 8 TB and one SSD of 1TB, but after checking my system configuration, I think I only have the two hard-drives :(

  
  Disk /dev/sda: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  Disklabel type: gpt
  Disk identifier: 2AA7A52A-3423-4201-887C-962F1AB2A7F6

Device Start End Sectors Size Type /dev/sda1 2048 999423 997376 487M EFI System /dev/sda2 999424 15628052479 15627053056 7.3T Linux RAID

Disk /dev/md0: 7.3 TiB, 8000915898368 bytes, 15626788864 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdb: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: F4F1DDD5-2B81-489C-9950-97BA881B80D8

Device Start End Sectors Size Type /dev/sdb1 2048 999423 997376 487M EFI System /dev/sdb2 999424 15628052479 15627053056 7.3T Linux RAID

Disk /dev/mapper/vg-root: 7.3 TiB, 7984914628608 bytes, 15595536384 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/mapper/vg-swap: 14.9 GiB, 15997075456 bytes, 31244288 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes

@Claybarn , this is what I get when I run only the Intan rec. controller
```bash
Item dropped at insertion point 0
Creating from description...
Sources::Intan Rec. Controller (2-1)
Rhd2000EvalBoardUsb3: Allocating 33.153 MBytes for USB buffer.
/home/segundo.martinez/git/plugin-GUI/Builds/Linux/build
---- Intan Technologies ---- Rhythm RHD2000 USB3 Controller v2.0 ----

FrontPanel DLL loaded.  Built: May 25 2017  14:06:51

Scanning USB for Opal Kelly devices...

Found 1 Opal Kelly device connected:
  Device #1: Opal Kelly XEM6310LX45 with serial number 1646000GH3

Attempting to connect to device '1646000GH3'
Opal Kelly device firmware version: 1.26
Opal Kelly device serial number: 1646000GH3
Opal Kelly device ID string: Opal Kelly XEM6310

Rhythm USB3 configuration file successfully loaded.

Initializing acquisition board.
Sample rate set to 30000
USB3 board mode enabled
Scanning
Sample rate set to 30000
Number of enabled data streams: 16
Checking for connected amplifier chips...
Number of enabled data streams: 2
Sample rate set to 30000
  Adding node to graph with ID number 100

Intan Rec. Controller updating settings.
Intan Rec. Controller setting num outputs to 67
Updating connections:

Signal chain: 0

Source node: Intan Rec. Controller.
     Connecting to audio and record nodes.
     No dest node.

Enabling processors...
Source node received enable signal
Expecting 67 channels.
Number of 16-bit words in FIFO: 0
Is eval board running: 0
Expecting blocksize of 22528 for 2 streams
Flushing FIFO.
FIFO count 0
Starting usb thread with buffer of 45056 bytes
Starting acquisition.
Filter Viewport disabled.
Creating new directory.
Sources/Intan Rec. Controller
Create subnodes with parameters
  Moving forward along signal chain.
  No processor found.
  End of chain.
Num Recording Processors: 1

Adding audio callback.
Start timestamp: 0

[JoseGuzman]

It is obviously a problem with the hard-disk. I will order a 1Tb SSD . Thank you very much for your help!

[jingjie-li]

Yes, that's the problem of our HDD, we are recording with 64 channels(+3AUXs) Cambridge neurotech's si-probe running with the latest ubuntu and open-ephys software. With an SSD that problem have got solved. Thanks, everyone!