erikhuck commented 3 months ago

This may not be as helpful information when child processes are used without showing the information for the main and child processes individually as described in #11. But the user can decide whether they want to see just the sum of all CPU percentages or that in addition to the more granular version. This stack overflow describes how this can be accomplished. We should explain in the documentation that a cpu usage of greater than 100% can occur if a process uses multiple threads across multiple cores.

hunter-moseley commented 3 months ago

This is actually very useful, since it can track full utilization of the CPU. Would recommend adding an option to divide by the number of CPU cores. This would then provide a percent utilization of the whole CPU across all cores.

On Tue, Mar 26, 2024 at 10:58 AM Erik Huckvale @.***> wrote:

This may not be as helpful information when child processes are used without showing the information for the main and child processes individually as described in #11 https://github.com/MoseleyBioinformaticsLab/gpu_tracker/issues/11. But the user can decide whether they want to see just the sum of all CPU percentages or that in addition to the more granular version. This stack overflow https://stackoverflow.com/questions/41206809/psutil-measuring-the-cpu-usage-of-a-specific-process describes how this can be accomplished. We should explain in the documentation that a cpu usage of greater than 100% can occur if a process uses multiple threads across multiple cores.

— Reply to this email directly, view it on GitHub https://github.com/MoseleyBioinformaticsLab/gpu_tracker/issues/13, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEP7BZ4I5QSUM5LQTIK6TDY2GEJLAVCNFSM6AAAAABFJDJDNSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDQNJQGA3TKMY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Hunter Moseley, Ph.D. -- Univ. of Kentucky Professor, Dept. of Molec. & Cell. Biochemistry / Markey Cancer Center / Institute for Biomedical Informatics / UK Superfund Research Center Not just a scientist, but a fencer as well. My foil is sharp, but my mind sharper still.

Email: @. (work) @. (personal) Phone: 859-218-2964 (office) 859-218-2965 (lab) 859-257-7715 (fax) Web: http://bioinformatics.cesb.uky.edu/ Address: CC434 Roach Building, 800 Rose Street, Lexington, KY 40536-0093

erikhuck commented 3 months ago

Possible data class

class MaxCPUPercentage:
   system_cpu_count: int
   system_cpu_percent: float
   system_average_cpu_percent: float # possibly
   main_cpu_percent: float
   main_normalized_cpu_percent: float
   descendent_cpu_percent: float
   descendent_average_cpu_percent: float
   combined_cpu_percent: float
   combined_average_cpu_percent: float
   main_n_threads: int
   descendent_n_threads: int
   combined_n_threads: int

psutil.cpu_count() returns the system-wide number of CPUs (cores).
- Don't worry about logical versus physical. Just accept the default return value.
The system-wide cpu_percent can be obtained via psutil.cpu_percent().
- We want to test this and determine what exactly this value represents
- Compare to the sum and average of the list of values returned when percpu is True.
Process.cpu_percent() provides the CPU utilization of a specific process.
- This value can exceed 100% if the process has multiple threads that are collectively using multiple cores.
- Can get an average CPU percent via p.cpu_percent() / psutil.cpu_count() where p is a Process.
It may also be helpful to list the max number of threads used by the process.
- This is obtained by Process.num_threads().

hunter-moseley commented 3 months ago

If the information is across CPU cores, then I would recommend the following key names: class MaxCPUUtilization: system_core_count: int system_core_percent: float system_cpu_percent: float # possibly main_core_percent: float main_cpu_percent: float descendent_core_percent: float descendent_cpu_percent: float combined_core_percent: float combined_cpu_percent: float main_n_threads: int descendant_n_threads: int combined_n_threads: int

hunter-moseley commented 3 months ago

Just realized one more important aspect of reporting CPU utilization. Both max and average results should be reported. Max will help indicate if the CPU is being fully utilized at any given time and average utilization will indicate how well the CPU is being utilized across time.

erikhuck commented 3 months ago

@hunter-moseley

What is the difference between cpu_percent and core_percent?
To account for collecting the average, should we rename the class to just CPUUtilization? And should we name the fields as so: system_max_core_percent, system_mean_core_percent, main_max_core_percent, main_mean_core_percent, etc.

hunter-moseley commented 3 months ago

A CPU has multiple CPU cores.

So, core_percent is in place of cpu_percent. And cpu_percent is in place of normalized_cpu_percent.

Yes, you can rename the class to CPUUtilization. Yes, use max and mean to distinguish between max and mean values.

On Wed, Apr 10, 2024 at 6:04 PM Erik Huckvale @.***> wrote:

@hunter-moseley https://github.com/hunter-moseley

What is the difference between cpu_percent and core_percent?

To account for collecting the average, should we rename the class to just CPUUtilization? And should we name the fields as so: system_max_core_percent, system_mean_core_percent, main_max_core_percent, main_mean_core_percent, etc.

— Reply to this email directly, view it on GitHub https://github.com/MoseleyBioinformaticsLab/gpu_tracker/issues/13#issuecomment-2048514706, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEP7B7FNBNGX5KID3ELM4DY4WZP5AVCNFSM6AAAAABFJDJDNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYGUYTINZQGY . You are receiving this because you were mentioned.Message ID: @.***>

-- Hunter Moseley, Ph.D. -- Univ. of Kentucky Professor, Dept. of Molec. & Cell. Biochemistry / Markey Cancer Center / Institute for Biomedical Informatics / UK Superfund Research Center Not just a scientist, but a fencer as well. My foil is sharp, but my mind sharper still.

Email: @. (work) @. (personal) Phone: 859-218-2964 (office) 859-218-2965 (lab) 859-257-7715 (fax) Web: http://bioinformatics.cesb.uky.edu/ Address: CC434 Roach Building, 800 Rose Street, Lexington, KY 40536-0093

erikhuck commented 3 months ago

@hunter-moseley I did an experiment and it is strongly evident that psutil.cpu_percent() is the same as numpy.mean(psutil.cpu_percent(percpu=True)). In other words, the mean percentage across all available CPUs. As I understand it, that would be the value for system_cpu_percent. So the question is, how would we like to calculate the value for system_core_percent? I'd think that would be the sum of all the core percentages as opposed to the mean? If that isn't very meaningful (perhaps because the value would possibly be more dependent on the number of cores in the system than anything else), we could alternatively exclude system_core_percent.

hunter-moseley commented 3 months ago

Use sum.

On Wed, Apr 10, 2024, 6:56 PM Erik Huckvale @.***> wrote:

@hunter-moseley https://github.com/hunter-moseley I did an experiment and it is strongly evident that psutil.cpu_percent() is the same as numpy.mean(psutil.cpu_percent(percpu=True)). In other words, the mean percentage across all available CPUs. As I understand it, that would be the value for system_cpu_percent. So the question is, how would we like to calculate the value for system_core_percent? I'd think that would be the sum of all the core percentages as opposed to the mean? If that isn't very meaningful (perhaps because the value would possibly be more dependent on the number of cores in the system than anything else), we could alternatively exclude system_core_percent.

— Reply to this email directly, view it on GitHub https://github.com/MoseleyBioinformaticsLab/gpu_tracker/issues/13#issuecomment-2048566132, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEP7BY6N4PRF3GAY24YYV3Y4W7S5AVCNFSM6AAAAABFJDJDNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYGU3DMMJTGI . You are receiving this because you were mentioned.Message ID: @.***>

erikhuck commented 3 months ago

So the final implementation should be?

@dclass.dataclass
class CPUUtilization:
    system_core_count: int
    system_max_core_percent: float = 0.
    system_max_cpu_percent: float = 0.
    main_max_core_percent: float = 0.
    main_max_cpu_percent: float = 0.
    descendent_max_core_percent: float = 0.
    descendent_max_cpu_percent: float = 0.
    combined_max_core_percent: float = 0.
    combined_max_cpu_percent: float = 0.
    system_mean_cpu_percent: float = 0.
    system_mean_core_percent: float = 0.
    main_mean_core_percent: float = 0.
    main_mean_cpu_percent: float = 0.
    descendent_mean_core_percent: float = 0.
    descendent_mean_cpu_percent: float = 0.
    combined_mean_core_percent: float = 0.
    combined_mean_cpu_percent: float = 0.
    main_n_threads: int = 0
    descendant_n_threads: int = 0
    combined_n_threads: int = 0

hunter-moseley commented 3 months ago

This looks complete.

On Wed, Apr 10, 2024, 7:26 PM Erik Huckvale @.***> wrote:

So the final implementation should be?

@dclass.dataclass class CPUUtilization: system_core_count: int system_max_core_percent: float = 0. system_max_cpu_percent: float = 0. main_max_core_percent: float = 0. main_max_cpu_percent: float = 0. descendent_max_core_percent: float = 0. descendent_max_cpu_percent: float = 0. combined_max_core_percent: float = 0. combined_max_cpu_percent: float = 0. system_mean_cpu_percent: float = 0. system_mean_core_percent: float = 0. main_mean_core_percent: float = 0. main_mean_cpu_percent: float = 0. descendent_mean_core_percent: float = 0. descendent_mean_cpu_percent: float = 0. combined_mean_core_percent: float = 0. combined_mean_cpu_percent: float = 0. main_n_threads: int = 0 descendant_n_threads: int = 0 combined_n_threads: int = 0

— Reply to this email directly, view it on GitHub https://github.com/MoseleyBioinformaticsLab/gpu_tracker/issues/13#issuecomment-2048594980, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEP7B4UUCA5AJT76VVLHMDY4XDEBAVCNFSM6AAAAABFJDJDNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYGU4TIOJYGA . You are receiving this because you were mentioned.Message ID: @.***>

erikhuck commented 3 months ago

Fixed by #30

MoseleyBioinformaticsLab / gpu_tracker

Add CPU percentage of processes to the measurements #13

-- Hunter Moseley, Ph.D. -- Univ. of Kentucky Professor, Dept. of Molec. & Cell. Biochemistry / Markey Cancer Center / Institute for Biomedical Informatics / UK Superfund Research Center Not just a scientist, but a fencer as well. My foil is sharp, but my mind sharper still.

Possible data class

-- Hunter Moseley, Ph.D. -- Univ. of Kentucky Professor, Dept. of Molec. & Cell. Biochemistry / Markey Cancer Center / Institute for Biomedical Informatics / UK Superfund Research Center Not just a scientist, but a fencer as well. My foil is sharp, but my mind sharper still.