Update tools with latest qualx and add CLI for train

[parthosa]

Fixes #1049.

We want to import the training module from QualX and update the prediction code with latest changes. This will be done in four steps.

Steps

1. Import the source code and pretrained models as it as. Exclude these from pytest and flake8 linters.
2. Make changes in the qualx files to connect them with tools.
3. Make changes in the tools to use the latest changes in qualx.
4. Add entry point for train by adding a new CMD for training.

Changes

• Step 1 and 2 were completed in #1060.
• This PR implements step 3 and 4.
• We introduce a base class QualXTool that extends RapidsTool for all qualx CMDs.

How to test/run?

We have three entry points with qualx

1. Python Qualification CMD:

spark_rapids qualification --platform <platform> --eventlogs </path/to/eventlogs> --estimation_model xgboost

2. Python Prediction CMD:

spark_rapids prediction --qual_output </path/to/qual_202406xxx> --prof_output </path/to/qual_202406xx> --output_folder </path/to/save/csv/files>

3. Python Train CMD (new):

Install optional dependencies for qualx training

cd user_tools
pip install -e .[qualx]

Train CMD:

spark_rapids train --dataset </path/to/dataset/files(s)> --model </path/to/save/trained/model> --output_folder </path/to/save/csv/files>

Note:

• More details on usage can be found in QualX docs.
• Instructions on how to download training dataset will be shared offline.

[leewyang]

Just to clarify things a bit per @amahussein's questions:

• We use the datasets json files to contain metadata about datasets, i.e. paths to eventlogs, appId to CPU and GPU mappings, etc.
• QUALX_DATA_DIR is the physical location of raw eventlogs on disk. This must be reachable via standard python I/O APIs. Typically, we keep the eventlogs in swiftstack, since they are fairly large, and sync to local disk.
• the QUALX_DATA_DIR env var allows users on different hosts to sync the eventlogs to their local disk at different locations. As such, this env variable is required as the prefix for all paths to eventlogs (in the dataset json files).
• this env variable is only used for training/evaluation of models (since it points to the location of the training/eval data).