Out of memory error after importing rt.dat files with telemetry

[tomvanbraeckel]

Using the latest Yamcs 4.10.9, importing around 19GB of rt.dat files with telemetry seemed to work and fill the archive nicely.

After the "incomingDir" had been emptied, the storage used on disk kept growing slightly, which I assume is nominal due to data processing.

But a few hours later, Yamcs Server ran out of RAM:

16:33:33.305 fsl-ops-rubi [3765] ArchiveFillerTask Aborting parameter archive filling due to imminent out of memory. Consider decreasing the maxSegmentSi
ze (current value is 5000).           
16:33:33.305 fsl-ops-rubi [3765] Processor [ParameterArchive-backfilling_1213] Processor ParameterArchive-backfilling_1213 quitting                      
16:33:33.305 fsl-ops-rubi [3765] Processor [ParameterArchive-backfilling_1213] Processor ParameterArchive-backfilling_1213 is out of business            
16:33:34.261 fsl-ops-rubi [27] BackFiller Parameter archive fillup for interval [2020-01-30T03:29:49.944Z-2020-01-30T05:49:38.552Z) aborted              
16:33:34.261 fsl-ops-rubi [27] BackFiller Starting parameter archive fillup for interval [2020-01-30T05:49:38.552Z-2020-01-30T08:09:27.160Z)             
16:33:34.261 fsl-ops-rubi [27] ProcessorFactory [ParameterArchive-backfilling_1214] Loading service ReplayService                  
16:33:34.262 fsl-ops-rubi [27] Processor Creating new processor 'ParameterArchive-backfilling_1214' of type 'ParameterArchive'   

16:33:34.261 fsl-ops-rubi [27] ProcessorFactory [ParameterArchive-backfilling_1214] Loading service ReplayService
16:33:34.262 fsl-ops-rubi [27] Processor Creating new processor 'ParameterArchive-backfilling_1214' of type 'ParameterArchive'
16:33:34.263 fsl-ops-rubi [27] MergeStream [merge1213] Starting merge stream
16:33:35.278 fsl-ops-rubi [3768] SequenceContainerProcessor Got java.lang.ArrayIndexOutOfBoundsException when extracting from the buffer of length 6768 bits, bitPosition 6768 entry: ParameterEntry position:433, container:FSL_EC_HK_SC locationInContainer:6768 from:containerStart, ParaName: FSL_EC_SC_Data413 paraType:IntegerDataType name:FSL_EC_SC_Data413 sizeInBits:32 signed:false encoding:IntegerDataEncoding[sizeInBits: 16, byteOrder: BIG_ENDIAN, encoding:UNSIGNED] aliases: {MDB:OPS Name=FSL_EC_SC_Data413, MDB:Pathname=\APM\USOC\MARS\FSL\ACQ\MEAS\FSL_EC_HK_SC\EC_SC_DATA413, MDB:SID=313225884}
16:33:35.279 _global [3768] DataEncodingDecoder Binary Parameter that does not start at byte boundary not supported. bitPosition: org.yamcs.utils.BitBuffer@26d083df
16:33:35.279 _global [3768] DataEncodingDecoder Binary Parameter that does not start at byte boundary not supported. bitPosition: org.yamcs.utils.BitBuffer@26d083df
16:33:35.279 fsl-ops-rubi [3768] ArchiveFillerTask Aborting parameter archive filling due to imminent out of memory. Consider decreasing the maxSegmentSize (current value is 5000).
16:33:35.279 fsl-ops-rubi [3768] Processor [ParameterArchive-backfilling_1214] Processor ParameterArchive-backfilling_1214 quitting
16:33:35.280 fsl-ops-rubi [3768] Processor [ParameterArchive-backfilling_1214] Processor ParameterArchive-backfilling_1214 is out of business
16:33:35.280 fsl-ops-rubi [27] BackFiller Parameter archive fillup for interval [2020-01-30T05:49:38.552Z-2020-01-30T08:09:27.160Z) aborted
16:33:36.240 fsl-ops-rubi [27] BackFiller Starting parameter archive fillup for interval [2020-01-30T08:09:27.160Z-2020-01-30T10:29:15.768Z)
16:33:36.240 fsl-ops-rubi [27] ProcessorFactory [ParameterArchive-backfilling_1215] Loading service ReplayService
16:33:36.240 fsl-ops-rubi [27] Processor Creating new processor 'ParameterArchive-backfilling_1215' of type 'ParameterArchive'
16:33:36.242 fsl-ops-rubi [27] MergeStream [merge1214] Starting merge stream
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/java_pid1099507.hprof ...
Heap dump file created [1685831205 bytes in 52.367 secs]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RdbTableReaderStream[tm_2451]"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-1221"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RdbTableReaderStream[pp_2452]"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-6"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-7-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RDBFactory-sync"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RDBFactory-sync"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-8-thread-1"

Here's a few screenshots of VisualVM:

Attached are some notes and I'll share the hprof file and Yamcs configuration privately.

[xpromache]

I think this was fixed by commit 96c7abd0b24278a9ac08c4fb47d102434182e110