search

EnSpec / hytools

Hyperspectral image processing library
GNU General Public License v3.0
56 stars 25 forks source link

Question - Ray parallelization #9

Closed serbinsh closed 2 years ago

serbinsh commented 2 years ago

Hello

I am experimenting with running hytools image corrections on our cluster and noticed that while it states running on 4 CPUs, most of the time the processes is rather sequential or low CPU usage? There are short periods where the number of CPUs selected (4) are used concurrently, but a lot of other times they are not?

Screen Shot 2022-03-09 at 2 35 32 PM

Screen Shot 2022-03-09 at 2 37 37 PM

Is this normal? It looks like the parallelization is running images on a CPU correct? Such that it would not help to have more CPUs availible than images?

serbinsh commented 2 years ago

Also FYI - here is a repo we are developing for running hytools jobs on our cluster. Still very much a WIP

https://github.com/TESTgroup-BNL/hytools_helpers

oldmanye007 commented 2 years ago

Basically, it is a coarse-grained parallelism. Yes, if available CPUs>number of images, only one CPU is paired to one image.

On Wed, Mar 9, 2022 at 1:40 PM Shawn P. Serbin @.***> wrote:

Hello

I am experimenting with running hytools image corrections on our cluster and noticed that while it states running on 4 CPUs, most of the time the processes is rather sequential or low CPU usage? There are short periods where the number of CPUs selected (4) are used concurrently, but a lot of other times they are not?

[image: Screen Shot 2022-03-09 at 2 35 32 PM] https://user-images.githubusercontent.com/3067369/157519388-aa035466-fe83-4ad1-9033-cffda0a954a5.png

[image: Screen Shot 2022-03-09 at 2 37 37 PM] https://user-images.githubusercontent.com/3067369/157519448-110caaed-b13b-4734-8573-ae4e0909d035.png

Is this normal? It looks like the parallelization is running images on a CPU correct? Such that it would not help to have more CPUs availible than images?

— Reply to this email directly, view it on GitHub https://github.com/EnSpec/hytools/issues/9, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEO2NVWJSUUJ4VHJRKYVO3U7D5BVANCNFSM5QKQXDQA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

serbinsh commented 2 years ago

OK thanks @oldmanye007