Error - System.OutOfMemoryException: Array dimensions exceeded supported range.

[ThomasLuypaert]

Describe the bug

When using ConcatenateIndexFiles with NoGaps rendering mode to create FCS for 672 1-min files representing 1 week of recording, with a framelength of 16,384 and acoustic indices calculated over 15 seconds, I get the following error:

System.OutOfMemoryException: Array dimensions exceeded supported range. (see screenshot)

Expected behavior

A FCS for the duration of one week. False Color Spectrograms were successfully derived using the same data with a lower framelength (2048, 4096 and 8192).

Diagnostics

• [x] I have followed the instructions for reporting a bug

• I have attached a log:

  log_20190724T134204Z.txt

• The Operating system I'm using: Windows 10 Home Version 1803

• The version of AnalysisPrograms.exe: v19.5.0.1

• The data used can be found: Data consists 672 1-min files, representing 1-min recording every 15-min for 24h/day for 1 week, at a sampling rate of 384 kHz. A framelength of 16,384 was used, and acoustic indices were calculated over 15 seconds.

• The command I used:

  D:/User/Software/AP/AnalysisPrograms.exe ConcatenateIndexFiles E:/Acoustic_Data/AM1_AIs/BatchIndicesOutput7/Week_1 -o E:/Acoustic_Data/AM1_AIs/BatchIndicesOutput7/Week_1/Output_Concatenate -d *.WAV* -ip D:/User/Software/AP/ConfigFiles/IndexPropertiesConfig.yml -fcs D:/User/Software/AP/ConfigFiles/SpectrogramFalseColourConfig.yml -f Acoustic_Test -z 0 -g NoGaps --draw-images -n --concatenate-everything --log-level 7 

• The config file I used:

Screenshots

Additional Details

Add any other context about the problem here.

[atruskie]

Thanks for the report (and for the attention to detail 😄). We've never processed data this large, in that particular dimension, before.

We have an internal re-design scheduled for the concatenate function (see #168) which should allow it to scale much better. Fixing this bug will be part of that.

In the meantime, I suggest running the concatenate in two parts. It is suboptimal, but it will work for now. The images can be placed next to each other for correct alignment. If you overlap the analysis a small bit temporally, then you paste the second image under the right edge of the first to hide the second y-axis produced.