search

ANTsX / ANTsR

R interface to the ANTs biomedical image processing library
https://antsx.github.io/ANTsR
Apache License 2.0
127 stars 35 forks source link

performance of ANTsR with respect to ANTs #197

Closed muratmaga closed 6 years ago

muratmaga commented 6 years ago

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

muschellij2 commented 6 years ago

I don't think they are the same defaults. You can see the defaults in

Also, I think ANTsR may use ITKGLOBALDEFAULT_NUMBER_OF_THREADS not ITK_GLOBAL_NUMBER_OF_THREADS.

I'd recommend giving a MCWE (https://stackoverflow.com/help/mcve) for this to actually debug.

John

On Thu, Jan 18, 2018 at 10:36 PM, muratmaga notifications@github.com wrote:

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBnrgiJzqt4pUQ5FMPjQa6RDRrZ1izMks5tMDf1gaJpZM4RkC9w .

muratmaga commented 6 years ago

Hi John,

Sorry, that was a typo in my end. It is indeed ITKGLOBALDEFAULT_NUMBER_OF_THREADS for both. And ANTsR was indeed using the multiple threads. I will prepare a sample dataset once retest on a dedicated system, and explicitly specifying the all parameters.

I just wanted to know if it is expected to get a performance hit running things through R and whether anyone have done a test of similar sorts.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: John Muschelli notifications@github.com Sent: Friday, January 19, 2018 9:38:24 AM To: ANTsX/ANTsR Cc: muratmaga; Author Subject: Re: [ANTsX/ANTsR] performance of ANTsR with respect to ANTs (#197)

I don't think they are the same defaults. You can see the defaults in

Also, I think ANTsR may use ITKGLOBALDEFAULT_NUMBER_OF_THREADS not ITK_GLOBAL_NUMBER_OF_THREADS.

I'd recommend giving a MCWE (https://stackoverflow.com/help/mcve) for this to actually debug.

John

On Thu, Jan 18, 2018 at 10:36 PM, muratmaga notifications@github.com wrote:

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBnrgiJzqt4pUQ5FMPjQa6RDRrZ1izMks5tMDf1gaJpZM4RkC9w .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FANTsX%2FANTsR%2Fissues%2F197%23issuecomment-359037423&data=02%7C01%7C%7C8a25373e48db40a356bb08d55f63a7d3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519803970627727&sdata=TNP3TzuIKKNVNC5V2D37ziT5IT8QoIT%2BEs8YNBTFwSg%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUTJFNXuKRG8wL9t7A7gt7LJ0ZKXQC8Aks5tMNMPgaJpZM4RkC9w&data=02%7C01%7C%7C8a25373e48db40a356bb08d55f63a7d3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519803970627727&sdata=dW8AFAgZdsCCo5zahPS88FPoypjT27CJbZ5Jptu%2Bw6s%3D&reserved=0.

stnava commented 6 years ago

not expected. both antsr and antspy call the same code that is used in ants - they are just wrappers. as indicated by @muschellij2 , you would have to explicitly set the parameters to be "the same". no need to test this on large 3D datasets; small 2D examples would be sufficient to verify.

brian

On Fri, Jan 19, 2018 at 12:52 PM, muratmaga notifications@github.com wrote:

Hi John,

Sorry, that was a typo in my end. It is indeed ITKGLOBALDEFAULT_NUMBER_OF_THREADS for both. And ANTsR was indeed using the multiple threads. I will prepare a sample dataset once retest on a dedicated system, and explicitly specifying the all parameters.

I just wanted to know if it is expected to get a performance hit running things through R and whether anyone have done a test of similar sorts.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: John Muschelli notifications@github.com Sent: Friday, January 19, 2018 9:38:24 AM To: ANTsX/ANTsR Cc: muratmaga; Author Subject: Re: [ANTsX/ANTsR] performance of ANTsR with respect to ANTs (#197)

I don't think they are the same defaults. You can see the defaults in

Also, I think ANTsR may use ITKGLOBALDEFAULT_NUMBER_OF_THREADS not ITK_GLOBAL_NUMBER_OF_THREADS.

I'd recommend giving a MCWE (https://stackoverflow.com/help/mcve) for this to actually debug.

John

On Thu, Jan 18, 2018 at 10:36 PM, muratmaga notifications@github.com wrote:

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197, or mute the thread https://github.com/notifications/unsubscribe-auth/ ABBnrgiJzqt4pUQ5FMPjQa6RDRrZ1izMks5tMDf1gaJpZM4RkC9w .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02. safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub. com%2FANTsX%2FANTsR%2Fissues%2F197%23issuecomment- 359037423&data=02%7C01%7C%7C8a25373e48db40a356bb08d55f63a7d3% 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519803970627727&sdata= TNP3TzuIKKNVNC5V2D37ziT5IT8QoIT%2BEs8YNBTFwSg%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook. com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth% 2FAUTJFNXuKRG8wL9t7A7gt7LJ0ZKXQC8Aks5tMNMPgaJpZM4RkC9w&data=02%7C01%7C% 7C8a25373e48db40a356bb08d55f63a7d3%7C84df9e7fe9f640afb435aaaaaaaa aaaa%7C1%7C0%7C636519803970627727&sdata=dW8AFAgZdsCCo5zahPS88FPoypjT27 CJbZ5Jptu%2Bw6s%3D&reserved=0.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197#issuecomment-359041036, or mute the thread https://github.com/notifications/unsubscribe-auth/AATyfrzZ0mX4pmPTA8zwB6gruHN9U0ZVks5tMNZ2gaJpZM4RkC9w .

dorianps commented 6 years ago

I used to do some comparisons like this. On some earlier versions of ANTsR I had a similar impression that ANTsR was taking longer. But at some point thing's got comparable. The engine for the registrations is exactly the same (provided you have updated ANTs and ANTsR versions). I would say that a maximum difference I would expect is 1hr in ants and 1hr20min in ANTsR, but not 10 times more. This said, I remember there is a slight difference in the number of iterations, sampling, or something else between antsRegistrationSyN.sh and antsRegistration('SyNCC'). In my opinion, however, the ANTsR output was slightly better than that from ANTs.

My advice is:

One last thing would be to check whether the two software manage the memory differently, but this might be unlikely to cause a problem in such a powerful system as yours.

Dorian

On Fri, Jan 19, 2018 at 12:52 PM, muratmaga notifications@github.com wrote:

Hi John,

Sorry, that was a typo in my end. It is indeed ITKGLOBALDEFAULT_NUMBER_OF_THREADS for both. And ANTsR was indeed using the multiple threads. I will prepare a sample dataset once retest on a dedicated system, and explicitly specifying the all parameters.

I just wanted to know if it is expected to get a performance hit running things through R and whether anyone have done a test of similar sorts.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: John Muschelli notifications@github.com Sent: Friday, January 19, 2018 9:38:24 AM To: ANTsX/ANTsR Cc: muratmaga; Author Subject: Re: [ANTsX/ANTsR] performance of ANTsR with respect to ANTs (#197)

I don't think they are the same defaults. You can see the defaults in

Also, I think ANTsR may use ITKGLOBALDEFAULT_NUMBER_OF_THREADS not ITK_GLOBAL_NUMBER_OF_THREADS.

I'd recommend giving a MCWE (https://stackoverflow.com/help/mcve) for this to actually debug.

John

On Thu, Jan 18, 2018 at 10:36 PM, muratmaga notifications@github.com wrote:

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197, or mute the thread https://github.com/notifications/unsubscribe-auth/ ABBnrgiJzqt4pUQ5FMPjQa6RDRrZ1izMks5tMDf1gaJpZM4RkC9w .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02. safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub. com%2FANTsX%2FANTsR%2Fissues%2F197%23issuecomment- 359037423&data=02%7C01%7C%7C8a25373e48db40a356bb08d55f63a7d3% 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519803970627727&sdata= TNP3TzuIKKNVNC5V2D37ziT5IT8QoIT%2BEs8YNBTFwSg%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook. com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth% 2FAUTJFNXuKRG8wL9t7A7gt7LJ0ZKXQC8Aks5tMNMPgaJpZM4RkC9w&data=02%7C01%7C% 7C8a25373e48db40a356bb08d55f63a7d3%7C84df9e7fe9f640afb435aaaaaaaa aaaa%7C1%7C0%7C636519803970627727&sdata=dW8AFAgZdsCCo5zahPS88FPoypjT27 CJbZ5Jptu%2Bw6s%3D&reserved=0.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197#issuecomment-359041036, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqafdt5-tYPs9EF7maAJw_F65tmYf7zks5tMNZ3gaJpZM4RkC9w .

muratmaga commented 6 years ago

Yeah I definitely prefer to work with ANTsR than ANTs for many reasons. Thanks for all the advice. I will try sometime soon again. M

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: dorianps notifications@github.com Sent: Friday, January 19, 2018 10:25:37 AM To: ANTsX/ANTsR Cc: muratmaga; Author Subject: Re: [ANTsX/ANTsR] performance of ANTsR with respect to ANTs (#197)

I used to do some comparisons like this. On some earlier versions of ANTsR I had a similar impression that ANTsR was taking longer. But at some point thing's got comparable. The engine for the registrations is exactly the same (provided you have updated ANTs and ANTsR versions). I would say that a maximum difference I would expect is 1hr in ants and 1hr20min in ANTsR, but not 10 times more. This said, I remember there is a slight difference in the number of iterations, sampling, or something else between antsRegistrationSyN.sh and antsRegistration('SyNCC'). In my opinion, however, the ANTsR output was slightly better than that from ANTs.

My advice is:

One last thing would be to check whether the two software manage the memory differently, but this might be unlikely to cause a problem in such a powerful system as yours.

Dorian

On Fri, Jan 19, 2018 at 12:52 PM, muratmaga notifications@github.com wrote:

Hi John,

Sorry, that was a typo in my end. It is indeed ITKGLOBALDEFAULT_NUMBER_OF_THREADS for both. And ANTsR was indeed using the multiple threads. I will prepare a sample dataset once retest on a dedicated system, and explicitly specifying the all parameters.

I just wanted to know if it is expected to get a performance hit running things through R and whether anyone have done a test of similar sorts.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: John Muschelli notifications@github.com Sent: Friday, January 19, 2018 9:38:24 AM To: ANTsX/ANTsR Cc: muratmaga; Author Subject: Re: [ANTsX/ANTsR] performance of ANTsR with respect to ANTs (#197)

I don't think they are the same defaults. You can see the defaults in

Also, I think ANTsR may use ITKGLOBALDEFAULT_NUMBER_OF_THREADS not ITK_GLOBAL_NUMBER_OF_THREADS.

I'd recommend giving a MCWE (https://stackoverflow.com/help/mcve) for this to actually debug.

John

On Thu, Jan 18, 2018 at 10:36 PM, muratmaga notifications@github.com wrote:

Hi,

I recently ran a test registering two large volumes using antsRegistrationSyN.sh and antsRegister on a vm with 44 vcpus

I specified 'CC' and 'SyN' for ANTsR explicitly and left everything else default. For the command line version everything was default. In both cases ITK_GLOBAL_NUMBER_OF_THREADS was set to 44, command line version that the -n 44 switch. R version took 10X longer than the shell script.

Are there expected performance losses running registration through R? I didn't check the number of iterations to be run (assumed will be same between ANTsR and command line, but may not be true). Another issue is, this was run on a cloud VM, so it is possible that other things might be running when the R version was executing. I will try to repeat the test on a dedicated system, but meanwhile would like to know if performance loss is expected when things run through R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197, or mute the thread https://github.com/notifications/unsubscribe-auth/ ABBnrgiJzqt4pUQ5FMPjQa6RDRrZ1izMks5tMDf1gaJpZM4RkC9w .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02. safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub. com%2FANTsX%2FANTsR%2Fissues%2F197%23issuecomment- 359037423&data=02%7C01%7C%7C8a25373e48db40a356bb08d55f63a7d3% 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519803970627727&sdata= TNP3TzuIKKNVNC5V2D37ziT5IT8QoIT%2BEs8YNBTFwSg%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook. com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth% 2FAUTJFNXuKRG8wL9t7A7gt7LJ0ZKXQC8Aks5tMNMPgaJpZM4RkC9w&data=02%7C01%7C% 7C8a25373e48db40a356bb08d55f63a7d3%7C84df9e7fe9f640afb435aaaaaaaa aaaa%7C1%7C0%7C636519803970627727&sdata=dW8AFAgZdsCCo5zahPS88FPoypjT27 CJbZ5Jptu%2Bw6s%3D&reserved=0.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ANTsX/ANTsR/issues/197#issuecomment-359041036, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqafdt5-tYPs9EF7maAJw_F65tmYf7zks5tMNZ3gaJpZM4RkC9w .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FANTsX%2FANTsR%2Fissues%2F197%23issuecomment-359049316&data=02%7C01%7C%7Cdba39c6a2c4c4fcfe2b308d55f6a0a1b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519831389601256&sdata=%2BwnwjHgZn3P1iq9kw3mxsmW%2BeTqG%2FnXDRdsrVINLK%2BU%3D&reserved=0, or mute the threadhttps://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUTJFIclG5KgrerjfZl2UK4hP7Mh_CzAks5tMN4hgaJpZM4RkC9w&data=02%7C01%7C%7Cdba39c6a2c4c4fcfe2b308d55f6a0a1b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636519831389601256&sdata=W7wvnh%2Bpg%2FMH%2BBSBIepDgAiCZGz%2BHNMZ27apFXmZqTI%3D&reserved=0.