Add qualification support for Photon jobs in the Python Tool

[parthosa]

Issue #251.

This PR introduces support for recommending Photon applications, using a separate strategy for categorizing them:

• Spark Runtime: Recommend apps with a speedup greater than 1.3x.
• Photon Runtime: Recommend apps with a speedup greater than 1x.

Additionally, the Small category for Photon applications is different from that of Spark-based applications:

• Spark Runtime: Apps with a speedup in the range of 1.3x to 2x are categorized as Small.
• Photon Runtime: Apps with a speedup in the range of 1x to 2x are categorized as Small.

Note

• Speedup Strategy is assigned on a per-app basis, enabling support for heterogeneous cases.
• Hence, if a user provides both Photon and Spark event logs, the Python Tool will apply separate strategy for each app based on its execution engine (Spark or Photon)

Output

• As this is a metadata property, for each app, included an entry sparkRuntime in app_metadata.json

  {
    "appId": "app-20240818062343-0000",
    "appName": "Databricks Shell",
    "eventLog": "file:/path/to/log/photon_eventlog",
    "sparkRuntime": "PHOTON",
    "estimatedGpuSpeedupCategory": "Not Recommended"
  }

Changes

Enhancements and New Features:

• tool_ctxt.py: Introduced a new method get_metrics_output_folder to fetch the metrics output directory.
• qualification-conf.yaml: Updated configuration to include new metrics subfolder and execution engine settings. [1] [2] [3] [4]
• enums.py: Added a new ExecutionEngine class to represent different execution engines.
• speedup_category.py: Introduced SpeedupStrategy class and refactored methods to accommodate execution engine-specific speedup strategies. [1] [2] [3] [4]

Refactoring and Utility Improvements:

• qualification.py: Added a helper method _read_qualification_metric_file to read metric files and _assign_execution_engine_to_apps to assign execution engines to applications.
• util.py: Added a utility method convert_df_to_dict to convert DataFrames to dictionaries.

Tests:

• event_log_processing.feature: Added new test scenarios to validate the execution engine assignment.
• e2e_utils.py and test_steps.py: Updated end-to-end test utilities to support new features. [1] [2] [3]

Follow Up

• 1417

[parthosa]

From offline discussions with @amahussein and @leewyang, moving the detection of runtime (Spark/Photon/Velox) to Scala.

This PR will be refactored afterwards.

[parthosa]

@amahussein

Is there another followup PR to change the QualX module to read the app_meta.json to decide whether this app is photon or not?

• After the changes in #1413, the logic to determine the runtime (whether Photon or not) has been moved to Scala.
• QualX module will now read the sparkRuntime column from the application_information.csv metric file to determine if the app is running on Photon.

I am concerned about how we can troubleshoot and validate app_meta.json. the wrapper reads the autotuner's output and copy some of the fields to that file in the upper level. With this PR, we are adding a new field derived from python logic. Later, we will hit a question "Where does each field come from?" (this becomes even more challenging if fields might be overridden by Python wrapper).

• Similarly, all values in app_meta.json will now be derived from Scala logic, with no Python logic involved.