Should set max fiber counts?

[soichih]

A user submitted with 2,000,000 total fibers (instead of 660,000 total) and generated the output track.tck with size of about 4G. When LIFE tried to process this, it ran out of memory.

I wonder if we should be able to set a reasonable max fiber counts to prevent other apps from failing. Or should other apps be able to handle large track.tck?

[francopestilli]

@soichih yes we should. @bcmcpher shall we set a max to 800,000? @ccaiafa we should fix this memory problem. I think you have done this on a branch didn't you

[ccaiafa]

Hi @francopestilli, yes you should use encode in the branch called Batch_Karst. It is only optimized for Karst, that is whay we didnt put in the master branch. I think @soichih already used it. @soichih let me know if you need help using the code in that branch

[francopestilli]

But the batch section should work any place correct?

What would we need to do to make it work outside of Karst? F-

On Feb 8, 2018, at 8:52 PM, Cesar Caiafa notifications@github.com wrote:

Hi @francopestilli https://github.com/francopestilli, yes you should use encode in the branch called Batch_Karst. It is only optimized for Karst, that is whay we didnt put in the master branch. I think @soichih https://github.com/soichih already used it. @soichih https://github.com/soichih let me know if you need help using the code in that branch

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brain-life/app-ensembletracking/issues/12#issuecomment-364310587, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBYc4pJVcfWnuCRhu0x-7zgzLLmC5wSks5tS6ThgaJpZM4R_Re6.

[ccaiafa]

Yes, you need to tune the MAXMEM variable by trial and error

[francopestilli]

OK good. The rest of the code is fine, identical to main?

On Feb 8, 2018, at 8:55 PM, Cesar Caiafa notifications@github.com wrote:

Yes, you need to tune the MAXMEM variable by trial and error

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brain-life/app-ensembletracking/issues/12#issuecomment-364311037, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBYc5Yv5y67iwXG5GgkzosQB6PZM-PHks5tS6V6gaJpZM4R_Re6.

[ccaiafa]

yes

[soichih]

@ccaiafa We've been using Batch_Karst branch encode for all environments. It works fine outside Karst, because you made it so that maxmem is calculated based on available memory at runtime.

[fe/feConnectomeEncoding.m]

MAXMEM = getenv('MAXMEM'); % read setting from enviroment in kb
if MAXMEM
    disp(['MAXMEM set to ',MAXMEM]);
    MAXMEM = str2num(MAXMEM);
elseif (isunix||ismac)
    disp('MAXMEM not set, need to calculate (UNIX or MacOS)')
    [~,out] = system('cat /proc/meminfo |grep MemFree');
    textcell = regexp(out,'\d*','Match');
    MAXMEM = str2num(textcell{1});
elseif ispc
    disp('MAXMEM not set, need to calculate (PC)')
    user = memory;
    MAXMEM = user.MaxPossibleArrayBytes/1024; % Max mem for arrays in Kb
end

Despite this, when it ran on carbonate with vmem=16gb, it ran into following ..

running feConnectomeInit

[feConnectomeInit]
 loading fiber from file: ../5a7c8d0f910651002a8561d3/track.tck
Reading Fiber Data for 1999998 fibers...

Files loaded: 
  dwi   = ../5a7c8d0e63458b0030c0da35/dwi_aligned_trilin_noMEC.nii.gz 
  bvecs = ../5a7c8d0e63458b0030c0da35/dwi_aligned_trilin_noMEC.bvecs 
  bvals = ../5a7c8d0e63458b0030c0da35/dwi_aligned_trilin_noMEC.bvals

[feBuildDictionaries] Computing demeaned difussivities Dictionary in a (360x360)-grid ...took: 0.004s.

[feConnectomeEncoding] Encoding connectome (building Phi, sparse tensor) ... MAXMEM set to 16000000
nNodesMax =15000000, Total number of nodes = 64391220,  number of batch computation = 5

Encoding batch  1
Encoding batch  2
Encoding batch  3
Encoding batch  4
Encoding batch  5
{Out of memory. Type HELP MEMORY for your options.

Error in unique>uniqueR2012a (line 313)
    groupsSortA = sortA(1:numRows-1,:) ~= sortA(2:numRows,:);

Error in unique (line 147)
        [varargout{1:nlhs}] = uniqueR2012a(varargin{1},logical(flaginds(1:5)));

Error in sptensor (line 227)
        [newsubs,junk,loc] = unique(subs,'rows');

Error in feConnectomeEncoding>concatenate_mode3 (line 95)
Phi1 = sptensor(subs1,vals1,s1);

Error in feConnectomeEncoding (line 72)
        Phi = concatenate_mode3(Phi, compute_Phi_batch(fe,fibers_range));

Error in feConnectomeInit (line 93)
fe = feConnectomeEncoding(fe);

Error in life (line 10)
fe = feConnectomeInit(aligned_dwi, ...

Error in main (line 25)
[ fe, out ] = life(config, aligned_dwi);
} 

It might be just that.. the MAXMEM calculation logic needs to be adjusted (to be slightly more conservative?)

[ccaiafa]

Thanks @soichih Yes, I tried to make it as more environment independent as possible but the MAXMEM calculation logics depend on a parameter tuned for Karst. Can you play around with the parameter and tune it to your environment? Let me know how it goes

[bcmcpher]

With how this is set up there is no way to define a maximum without arbitrarily ignoring iterations of parameters going into the ensemble. I wrote it to explicitly force the user to know how many streamlines were included from each parameter combination. The old method routinely estimated <10,000 streamlines per lmax/curv/track combination due to users requesting a large number of parameters w/o increasing the total. The "equal" number divided out from the 500k requested was never reported, so how many were generated for each combination was unknown (it also rarely equaled 500k).

The best way to solve this is to reduce the number of lmax/curv/track types requested by default or to simply error out if the # of tracks exceeds 800k and tell them to choose fewer parameters.

On Thu, Feb 8, 2018 at 9:27 PM, Cesar Caiafa notifications@github.com wrote:

Thanks @soichih https://github.com/soichih Yes, I tried to make it as more environment independent as possible but the MAXMEM calculation logics depend on a parameter tuned for Karst. Can you play around with the parameter and tune it to your environment? Let me know how it goes

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brain-life/app-ensembletracking/issues/12#issuecomment-364316211, or mute the thread https://github.com/notifications/unsubscribe-auth/ANRr_aY-ep8pj3gGrLSY8oRxrpmjjLh6ks5tS60WgaJpZM4R_Re6 .