Analysis of Emission Pipeline Top 20% and Bottom 80%

[TeachMeTW]

Introduces two distinct performance analysis methods to evaluate function-level metrics within our emission dataset. The objective is to identify which functions significantly impact performance and which do not, enabling targeted optimizations and improvements.

Analysis Types

1. Individual Entry Categorization

• Purpose: Categorizes each individual data.reading entry into Top 20% or Bottom 80% based on the 80th percentile within each data.name group.
• Use Case: Identifies specific high-impact executions of functions, allowing for pinpointing problematic instances.

Features

• Exclusions: Specific functions are excluded as they are parent of smaller functions and provide no insights:
  • TRIP_SEGMENTATION/segment_into_trips
  • TRIP_SEGMENTATION/segment_into_trips_dist/loop
• Sorting: Both Top 20% and Bottom 80% categories are sorted in descending order of data.reading for easy identification of high-impact entries.

---

2. Aggregated Entry Categorization

• Purpose: Aggregates data.reading metrics (both sum and mean) for each data.name and categorizes the aggregated values into Top 20% and Bottom 80% based on their respective 80th percentiles.
• Use Case: Determines which functions are consistently resource-intensive on average or cumulatively, providing a broader view of performance impact.

Features

• Aggregation Types:
  • Sum Aggregation: Total data.reading per function.
  • Mean Aggregation: Average data.reading per function.
• Sorting: Both Top 20% and Bottom 80% categories are sorted in descending order of aggregated data.reading.

---

bottom80_function_level_individual_sorted.csv bottom80_function_level_mean_sorted.csv bottom80_function_level_sum_sorted.csv top20_function_level_individual_sorted.csv top20_function_level_mean_sorted.csv top20_function_level_sum_sorted.csv

[TeachMeTW]

From both mean and sum, the same functions show up in the top 20% and bottom 80% albeit a different order. I will go ahead and prune them from the pipeline.