search

betterscientificsoftware / bssw.io

Better Scientific Software Homepage
https://bssw.io
Other
140 stars 90 forks source link

Software (Quality) Metrics #653

Open bernhold opened 3 years ago

bernhold commented 3 years ago

In the spirit of "all metrics are wrong, but some are useful", perhaps we should have some articles about widely used software metrics?

bernhold commented 3 years ago

Todd Gamblin recently mentioned https://github.com/ossf/criticality_score/blob/main/Quantifying_criticality_algorithm.pdf which is not the first thing I would think of as a software quality metric, but it might also fit this general idea.

markcmiller86 commented 3 years ago

Found this 2006 paper with specific metrics and thresholds...

http://www.mccabe.com/pdf/McCabe%20Recommended%20Approach%20to%20Code%20Reviews.pdf

rinkug commented 3 years ago

Good idea! Need some good resources to point to.

brnorris03 commented 3 years ago

The Promise public repository has many datasets with references to papers defining the metrics. http://promise.site.uottawa.ca/SERepository/datasets-page.html

rinkug commented 1 year ago

Another article: https://www.software.ac.uk/blog/2017-05-09-software-metrics-why-and-how

markcmiller86 commented 1 year ago

@bernhold I asked ChatGPT for some info on this...

High-Performance Computing (HPC) and Computational Science and Engineering (CSE) communities rely on a variety of software metrics to assess and improve the performance, reliability, and efficiency of their applications and systems. Some of the most widely used and useful software metrics within these communities include:

  1. Execution Time: This is one of the most basic metrics, measuring the time it takes for a program to run. It's crucial in HPC/CSE for assessing the performance of parallel algorithms and evaluating optimizations.

  2. Speedup and Scalability: Speedup measures the performance improvement of a parallel program compared to its serial counterpart, while scalability evaluates how well the program's performance scales with increasing resources (e.g., more processors). Metrics like strong scaling and weak scaling are commonly used.

  3. Parallel Efficiency: This metric assesses how efficiently a parallel program uses the available resources. It's often expressed as a ratio of the speedup to the number of processors, indicating how well the program scales.

  4. FLOP/s (Floating-Point Operations per Second): FLOP/s is a metric used to measure the computational throughput of HPC applications. It helps evaluate the efficiency of algorithms and their implementation in terms of floating-point operations.

  5. Memory Usage: Monitoring memory consumption is critical, as inadequate memory can lead to performance bottlenecks. Metrics like peak memory usage and memory bandwidth utilization are commonly tracked.

  6. I/O Performance: For many scientific simulations, data storage and retrieval can be a significant bottleneck. Metrics like IOPS (Input/Output Operations Per Second), throughput, and latency are crucial for assessing I/O performance.

  7. Cache Performance: Evaluating cache hit rates and cache misses can provide insights into memory access patterns and guide optimizations to reduce cache misses.

  8. Load Balancing: In parallel computing, ensuring that work is evenly distributed among processors is essential for efficient execution. Load balancing metrics measure the degree to which work is distributed uniformly across processors.

  9. Communication Overhead: In distributed computing environments, communication between nodes can be a major performance bottleneck. Metrics like message passing time and network bandwidth usage help assess communication efficiency.

  10. Energy Efficiency: As power consumption is a concern in large HPC clusters, energy efficiency metrics help assess the computational work done per unit of energy consumed.

  11. Software Complexity Metrics: These metrics evaluate the complexity of the codebase, such as lines of code, cyclomatic complexity, or code duplication. Lower complexity often leads to easier maintenance and improved performance.

  12. Reliability Metrics: Metrics related to fault tolerance and error handling, such as mean time between failures (MTBF) and mean time to recovery (MTTR), are important for ensuring system resilience.

  13. Code Profiling: Profiling tools generate metrics on code execution, identifying performance bottlenecks, hotspots, and areas for optimization. Common profiling tools include gprof, perf, and VTune.

  14. Code Coverage: Measuring code coverage helps assess the thoroughness of testing. It ensures that most parts of the code are exercised by test cases.

  15. Software Quality Metrics: Metrics related to code quality, such as code reviews, bug tracking, and code maintainability assessments, help maintain robust and reliable software.

These metrics play a crucial role in optimizing and assessing the performance of software in HPC and CSE applications. The choice of metrics depends on the specific goals and characteristics of the application and system being evaluated.

Many of the above are performance metrics and those are interesting but is that what you were thinking?

rinkug commented 1 year ago

@bernhold I asked ChatGPT for some info on this...

High-Performance Computing (HPC) and Computational Science and Engineering (CSE) communities rely on a variety of software metrics to assess and improve the performance, reliability, and efficiency of their applications and systems. Some of the most widely used and useful software metrics within these communities include:

  1. Execution Time: This is one of the most basic metrics, measuring the time it takes for a program to run. It's crucial in HPC/CSE for assessing the performance of parallel algorithms and evaluating optimizations.
  2. Speedup and Scalability: Speedup measures the performance improvement of a parallel program compared to its serial counterpart, while scalability evaluates how well the program's performance scales with increasing resources (e.g., more processors). Metrics like strong scaling and weak scaling are commonly used.
  3. Parallel Efficiency: This metric assesses how efficiently a parallel program uses the available resources. It's often expressed as a ratio of the speedup to the number of processors, indicating how well the program scales.
  4. FLOP/s (Floating-Point Operations per Second): FLOP/s is a metric used to measure the computational throughput of HPC applications. It helps evaluate the efficiency of algorithms and their implementation in terms of floating-point operations.
  5. Memory Usage: Monitoring memory consumption is critical, as inadequate memory can lead to performance bottlenecks. Metrics like peak memory usage and memory bandwidth utilization are commonly tracked.
  6. I/O Performance: For many scientific simulations, data storage and retrieval can be a significant bottleneck. Metrics like IOPS (Input/Output Operations Per Second), throughput, and latency are crucial for assessing I/O performance.
  7. Cache Performance: Evaluating cache hit rates and cache misses can provide insights into memory access patterns and guide optimizations to reduce cache misses.
  8. Load Balancing: In parallel computing, ensuring that work is evenly distributed among processors is essential for efficient execution. Load balancing metrics measure the degree to which work is distributed uniformly across processors.
  9. Communication Overhead: In distributed computing environments, communication between nodes can be a major performance bottleneck. Metrics like message passing time and network bandwidth usage help assess communication efficiency.
  10. Energy Efficiency: As power consumption is a concern in large HPC clusters, energy efficiency metrics help assess the computational work done per unit of energy consumed.
  11. Software Complexity Metrics: These metrics evaluate the complexity of the codebase, such as lines of code, cyclomatic complexity, or code duplication. Lower complexity often leads to easier maintenance and improved performance.
  12. Reliability Metrics: Metrics related to fault tolerance and error handling, such as mean time between failures (MTBF) and mean time to recovery (MTTR), are important for ensuring system resilience.
  13. Code Profiling: Profiling tools generate metrics on code execution, identifying performance bottlenecks, hotspots, and areas for optimization. Common profiling tools include gprof, perf, and VTune.
  14. Code Coverage: Measuring code coverage helps assess the thoroughness of testing. It ensures that most parts of the code are exercised by test cases.
  15. Software Quality Metrics: Metrics related to code quality, such as code reviews, bug tracking, and code maintainability assessments, help maintain robust and reliable software.

These metrics play a crucial role in optimizing and assessing the performance of software in HPC and CSE applications. The choice of metrics depends on the specific goals and characteristics of the application and system being evaluated.

Many of the above are performance metrics and those are interesting but is that what you were thinking?

Looks quite comprehensive!