search

rbeckman-nextgen / test-mc3

0 stars 0 forks source link

File Reader - improve memory resource management for loading large number of files #2156

Open rbeckman-nextgen opened 4 years ago

rbeckman-nextgen commented 4 years ago

Occasionally when there exist large number of files to read from a file reader directory that the available system memory can handle, Java GC heap size error occurs and no channel is deployed as a result. This also applies to processing queued messages, which would exist on a file system.

I think we can improve the listFiles() method in the FileConnection class to better handle this.

Although it would've been an ideal solution, I do not believe there's a way to limit the number of files to read. Currently the listFiles() returns an arraylist of File objects of all existing files. I'd suggest using list() method instead, which returns an arraylist of String and iterate through the list and create a File object for each file to more efficiently utilize the memory.

Imported Issue. Original Details: Jira Issue Key: MIRTH-2233 Reporter: chrisr Created: 2012-10-01T15:10:18.000-0700

rbeckman-nextgen commented 4 years ago

We may be able to do this for the File mode by using Java NIO directory streams. However we'd also need to add a "None" option to the sorting, because when limiting an ls we can't sort.

Imported Comment. Original Details: Author: narupley Created: 2019-06-24T14:59:30.000-0700