Matlab Crashes while Writing

[Paddy159]

Hello, i want to write long eeg recordings to a .adidat file. However, it crashes during the stopRecording command. You can find the code I'm using below (it generates example data, but the same happens with the acutal data):

      file_path = 'path_to_\test.adidat';
      fw = adi.createFile(file_path);
      fs1 = 250;
      chan = 1;
      pres_chan = fw.addChannel(chan,'pressure',fs1,'cmH20');
      fs2 = 250;
      chan = 2;
      pres2_chan = fw.addChannel(chan,'emg',fs2,'mV');
      fw.startRecord();
      y1(1:65000000) = 1.23456789;
      pres_chan.addSamples(y1);
      y2(1:65000000) = 1.23456789;
      pres2_chan.addSamples(y2);
     fw.stopRecord(); %<-- here it crashes

If I am writing the same (or twice) the number of datapoints into one channel, everything works fine. I attached the Matlab crash report. Do you have any idea what might cause the crash?

crashlog.txt

Thanks a lot for your amazing work! Best, Patrick

[Paddy159]

I can provide additional information ... I played around with the commands and if I dont run fw.stopRecord(); but save the file, Matlab does not crash.

When I create 3 channels and fill them with data, the first channel is fine, the second channel is corrupted after ~ 20h recording, and the third channel is completely corrupted. The code I used is here:

file_path = 'path_to\test.adidat';
fw = adi.createFile(file_path);
fs1 = 250;
chan = 1;
pres_chan = fw.addChannel(chan,'pressure',fs1,'V');
fs2 = 250;
chan = 2;
pres2_chan = fw.addChannel(chan,'emg',fs2,'V');

fs3 = 250;
chan = 3;
pres3_chan = fw.addChannel(chan,'data3',fs3,'V');
fw.startRecord();
y1(1:65000000) = 1.23456789;
pres_chan.addSamples(y1);
y2(1:65000000) = 1.23456789;
pres2_chan.addSamples(y2);
y3(1:65000000) = 1.23456789;
pres3_chan.addSamples(y3);
%fw.stopRecord(); %<-- here it crashes
fw.save
fw.close
clear fw emg_chan pres_chan pres2_chan pres3_chan

The resulting adidat file is attached.

Best, Patrick file.zip

[JimHokanson]

So in my testing with 2 channels the crash occurs around ~2.5e7 samples, but it is not consistent (I think). At first I thought it might be an overflow error, but 2.5e7, even for 2 channels with 4 bytes isn't close to the 32 bit max (and is way past the 16 bit max). Interestingly, with 1 channel I tested up to 1e8 samples and had no issue (saving a constant, so compression is easy). That being said, I know internally there is some compression going on so my best guess is that with the compression the system needs some (variable) multiple of the input bytes in memory AND it is probably using a 32 bit max somewhere, thus causing the crash.

I think what ADInstruments does for normal files (at least with Labchart 8) is that it periodically flushes to disk and then on saving moves the file from a temporary location to the real location. This gives the illusion of not saving while ensuring that it doesn't have to hold everything in memory.

The workaround, that I tested, is to simply save periodically when adding samples (see code example below). For large vectors this means writing in pieces. I could add on some functionality that tracks the number of samples being written and throws an error if too many are written before saving. This wouldn't be too hard although I would need to figure out how it handles channels with different sampling rates (does it replicate slower channels, thus increasing the number of bytes in memory?).

More importantly, is there a better interface for simply dumping a set of variables (table?) to disk? Or put another way, there obviously should be, but what should that interface look like? Ideally in this case the internals would then simply handle the periodic saving. I'm open to suggestions. That being said, I'm not using this functionality at all so with a workaround available it is low priority for me -- although I'd be open to a pull request (or emailing a file would be fine too).

sizes = 5e7:2000000:1e8;

for i = 1:length(sizes)
    n = sizes(i);
    fprintf(2,'%d\n',n);
    file_path = 'D:\test.adidat';
    if exist(file_path,'file')
        delete(file_path)
    end
    fw = adi.createFile(file_path);
    fs1 = 250;
    chan = 1;
    pres_chan = fw.addChannel(chan,'pressure',fs1,'cmH20');
    fs2 = 250;
    chan = 2;
    pres2_chan = fw.addChannel(chan,'emg',fs2,'mV');
    fw.startRecord();
    max_write = 5e6;
    %This is a bit of a hack but illustrates the point
    n_writes = n/max_write; 
    for j = 1:n_writes
    y1(1:max_write) = 1:max_write;
    pres_chan.addSamples(y1);
    y2(1:max_write) = max_write;
    pres2_chan.addSamples(y2);
    fw.save();  %This is the key line
    end
    fw.stopRecord();
    fw.saveAndClose();
    clear fw
    clear pres_chan
    clear pres2_chan
end

[Paddy159]

Thank you very much for your instant help! Now it works like a charm :)

Can you help me with two other questions?

1) Can i define and pass a custom starting time/date? I want to set the starting time/date to the actual recording not to "now". 2) I there a possibility to change the range of the channels? Its always -12 to +12. Everything above/below is shown as "out of range"

Best, Patrick

[JimHokanson]

Hmm, so at one point during debugging I temporarily set the limits to +-12 but it should be back at -inf and +inf. If you want more control over that (something more specific than Inf) I would need to add a few things. Note the data are internally stored as a single so Inf is supported

Regarding the start of the record try something like this (haven't tested):

start_time = datenum(2001,12,19,18,0,0);
fw.startRecord('trigger_time',start_time); 

[Paddy159]

Thank you very much for your reply. I downloaded the acutal code, however I get an error when i run it:

Unrecognized field name "record_to_copy".

Error in adi.file_writer/startRecord (line 360)
            if ~isempty(in.record_to_copy)

Error in ImportData (line 116)
    fw.startRecord();

I commented out line 360-363, this helps to prevent the error. However, I received another error which I was not able to fix:

Error using adi.sl.in.processVarargin>processVararginHelper
Bad variable names given in input structure:
---------------------------------
 record_spacing
--------------------------------------

Error in adi.sl.in.processVarargin (line 89)
[in,extras] = processVararginHelper(in,v,c,false);

Error in adi.sdk.startRecord (line 359)
           in = adi.sl.in.processVarargin(in,varargin);

Error in adi.file_writer/startRecord (line 381)
            adi.sdk.startRecord(obj.data_writer_h,in);

Do you have any Idea?

Best, Patrick

[JimHokanson]

Well that's what you get when you write code and don't actually test it. I've uploaded some new code so it should all be set. I've also removed the dependency of my standard library code. Please let me know if that works. Oh, and you no longer need to convert from datetime to datenum.

[Paddy159]

Thank you so much Jim, everything works great now!

Best, Patrick

[Paddy159]

Hi Jim, when I'm writing data to one file, i noticed that the file-size increased exponentially when i doubled the length of the data. Is that the normal behaviour? I used the code you wrote:

sizes = 5e7:2000000:1e8;

for i = 1:length(sizes)
    n = sizes(i);
    fprintf(2,'%d\n',n);
    file_path = 'D:\test.adidat';
    if exist(file_path,'file')
        delete(file_path)
    end
    fw = adi.createFile(file_path);
    fs1 = 250;
    chan = 1;
    pres_chan = fw.addChannel(chan,'pressure',fs1,'cmH20');
    fs2 = 250;
    chan = 2;
    pres2_chan = fw.addChannel(chan,'emg',fs2,'mV');
    fw.startRecord();
    max_write = 5e6;
    %This is a bit of a hack but illustrates the point
    n_writes = n/max_write; 
    for j = 1:n_writes
    y1(1:max_write) = 1:max_write;
    pres_chan.addSamples(y1);
    y2(1:max_write) = max_write;
    pres2_chan.addSamples(y2);
    fw.save();  %This is the key line
    end
    fw.stopRecord();
    fw.saveAndClose();
    clear fw
    clear pres_chan
    clear pres2_chan
end

Best, Patrick

[JimHokanson]

What file sizes are you seeing? When I run the above code the first file starts at about 7 MB and the last file (double the samples) is about 14 MB (double the size). The system is using compression so if you add something much more complicated for a section the file size is going to increase much more than if you add something simple.