Error estimating memory requirements on very low resolution inputs

[pipitone]

I created an extremely low resolution (step size 10) test case to play with this by resampling the winterburn atlases like so:

autocrop -isostep 10 T1.mnc input/atlas/subject1_t1.mnc

I'm using the atlas images as templates and subjects as well.

mb.sh b0rks when computing the memory requirements for qbatch by returning a negative number, and then failing when calling qbatch:

$ mb.sh run
2016-07-29 01:01:01 UTC [     info] Found:
2016-07-29 01:01:01 UTC [     info]   5 atlases in input/atlas
2016-07-29 01:01:01 UTC [     info]   1 labels in input/atlas
2016-07-29 01:01:01 UTC [     info]   5 templates in input/template
2016-07-29 01:01:01 UTC [     info]   5 subjects in input/subject
2016-07-29 01:01:01 UTC [     info]   0 models in input/models
2016-07-29 01:01:01 UTC [     info] Progress:
2016-07-29 01:01:01 UTC [     info]   0 of 25 atlas-template registrations completed
2016-07-29 01:01:01 UTC [     info]   0 of 20 template-subject registrations completed
2016-07-29 01:01:01 UTC [     info]   0 of 125 resample labels completed
2016-07-29 01:01:01 UTC [     info]   0 of 5 voted labels completed
2016-07-29 01:01:01 UTC [     info] Checking Resolution of First Atlas
2016-07-29 01:01:02 UTC [     info]   Found average: 10
2016-07-29 01:01:02 UTC [     info] Checking Resolution of First Template
2016-07-29 01:01:02 UTC [     info]   Found average: 10
2016-07-29 01:01:02 UTC [     info] Checking Resolution of First Subject
2016-07-29 01:01:02 UTC [     info]   Found average: 10
2016-07-29 01:01:02 UTC [   notice] MAGeTbrain makes no attempt to find the maximum resolution file, if you have mixed resolutions, make your highest resoluition file the first file
2016-07-29 01:01:02 UTC [     info] Computing Atlas to Template Registrations
usage: qbatch [options] <command_file>
qbatch: error: argument --mem: expected one argument
2016-07-29 01:01:02 UTC [     info] Cleaning up. Done

Perhaps lower bounds on memory/walltime estimates?

[gdevenyi]

See voxel based memory and time estimate branch.

[gdevenyi]

https://github.com/CobraLab/antsRegistration-MAGeT/tree/voxel-based-memory-time

Fun fact, ants is nicely linear with voxel count

[gdevenyi]

Also see https://github.com/gdevenyi/antsRegistration-benchmarking for source of my data

[pipitone]

Okay. So I should use that branch?

On Jul 28, 2016, at 9:24 PM, "Gabriel A. Devenyi" notifications@github.com wrote:

See voxel based memory and time estimate branch.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[pipitone]

Okay, I tried that branch. Looks nice, but I get a qsub error when the memory is < 1G (apparently our version of pbs doesn't like non-integer sizes for memory (man pbs_resources). Not sure if this is a qbatch problem or an mb issue. ;-)

[gdevenyi]

Okay, good to know.

I think putting a floor of 1GB of ram is probably reasonable anyways, even if we end up using less. I can roll that in after the maths transition to python.

I wouldn't call it a qbatch problem. I think we decided that qbatch shouldn't check the format of inputs for the batch system, it's the batch system that should error out.

[pipitone]

I think we're good with the latest merge.

[gdevenyi]

Do you think a memory floor still makes sense?

[pipitone]

I snuck one in ;-)

https://github.com/CobraLab/antsRegistration-MAGeT/blob/master/bin/stages.sh#L30

[gdevenyi]

Oh :+1:

I looked at that and was confused why there was a max call in there.... makes sense now.