Finalise Information Value function and miscellaneous

[martinctc]

Summary of IV functionality:

This branch finalizes changes to the Information Value (IV) functions.

Information Value (IV) is a powerful methodology that provides a measure of the predictive power of an individual independent variable in relation to the dependent variable. In the context of Viva Insights, independent variables could be a collaboration metric (e.g. Emails sent, 1:1 meeting time with managers), whereas a dependent variable could be a categorical variable indicating whether a person is engaged, a top performer, or at risk of attrition - likely provided through a survey.

This pull request also updates the package version to v0.3.0 and is released to PyPI.

Changes

The changes made in this PR are:

1. New Information value feature with create_IV function. This in turn calls a number of other functions to generate statistics and visualizations around Information Value (IV) and Weight of Evidence (WoE).
2. New visualization with create_bar_asis function, which can be leveraged outside of the Information Value context.

Checks

• [x] All checks pass (run python -m unittest discover -s tests -f locally)
• [x] Versions are updated in pyproject.toml, setup.py, and docs/conf.py (if applicable)
• [x] NEWS.md has been updated

Function Summary Related to Information Value (IV)

• _ptest(data, outcome, behavior, paired=False): This function performs a statistical test (Wilcoxon signed-rank test or rank-sum test) to compare two groups of data based on an outcome variable.

  • Parameters:
    • data: Pandas DataFrame containing the dataset.
    • outcome: Name of the outcome variable.
    • behavior: List of predictor variables.
    • paired: Boolean indicating whether the test is paired or unpaired (default is False).
    • Returns: Pandas DataFrame with p-values for each predictor variable.

• _calculateIV(data, outcome, predictor, bins): Calculates Information Value (IV) for a given predictor variable with respect to the outcome variable.

  • Parameters:
  • data: Pandas DataFrame containing the dataset.
  • outcome: Name of the outcome variable.
  • predictor: Name of the predictor variable.
  • bins: Number of bins to use for discretizing the predictor variable.
  • Returns: Pandas DataFrame with IV values for the predictor variable.

• _mapIV(data, outcome, predictors=None, bins=5): Maps IV values for all predictor variables in the dataset. Parameters:

  • data: Pandas DataFrame containing the dataset.
  • outcome: Name of the outcome variable.
  • predictors: List of predictor variables (default is None, which selects all numeric columns).
  • bins: Number of bins to use for discretizing the predictor variables.
  • Returns: Dictionary containing individual IV tables and a summary table sorted by IV values.

Function Architecture

• _create_IV(data, predictors=None, outcome=None, bins=5, siglevel=0.05, exc_sig=False, returntype="plot"):
  • This is the main function that orchestrates the calculation and visualization of IV values.
  • _It first filters the dataset based on the significance level of predictor variables using ptest.
  • _Then, it calculates IV values for the selected predictor variables using mapIV.
  • _Depending on the returntype, it either returns a summary table, individual IV tables, plots, or a combination of these.

Function Relationships

• _createIV calls:
  • _ptest to filter the dataset based on the significance level.
  • _mapIV to calculate IV values for selected predictor variables.
  • _Depending on the return_type, it may call plot_WOE or create_barasis to visualize the results.

4. Overall Description:

• The code aims to analyze the predictive power of predictor variables using IV and related statistics.
• It provides functions to calculate IV, perform statistical tests, and visualize the results.
• _The main function createIV serves as a controller function that manages the entire process from data filtering to visualization, offering flexibility in terms of the type of output desired.

[sachinstl]

Summary of IV functionality: 1. Function Summary Related to Information Value (IV):

• _ptest(data, outcome, behavior, paired=False): This function performs a statistical test (Wilcoxon signed-rank test or rank-sum test) to compare two groups of data based on an outcome variable.

  • Parameters:
    • data: Pandas DataFrame containing the dataset.
    • outcome: Name of the outcome variable.
    • behavior: List of predictor variables.
    • paired: Boolean indicating whether the test is paired or unpaired (default is False).
    • Returns: Pandas DataFrame with p-values for each predictor variable.

• _calculateIV(data, outcome, predictor, bins): Calculates Information Value (IV) for a given predictor variable with respect to the outcome variable.

  • Parameters:
  • data: Pandas DataFrame containing the dataset.
  • outcome: Name of the outcome variable.
  • predictor: Name of the predictor variable.
  • bins: Number of bins to use for discretizing the predictor variable.
  • Returns: Pandas DataFrame with IV values for the predictor variable.

• _mapIV(data, outcome, predictors=None, bins=5): Maps IV values for all predictor variables in the dataset. Parameters:

  • data: Pandas DataFrame containing the dataset.
  • outcome: Name of the outcome variable.
  • predictors: List of predictor variables (default is None, which selects all numeric columns).
  • bins: Number of bins to use for discretizing the predictor variables.
  • Returns: Dictionary containing individual IV tables and a summary table sorted by IV values.

2. Function Architecture:

• _create_IV(data, predictors=None, outcome=None, bins=5, siglevel=0.05, exc_sig=False, returntype="plot"):
  • This is the main function that orchestrates the calculation and visualization of IV values.
  • _It first filters the dataset based on the significance level of predictor variables using ptest.
  • _Then, it calculates IV values for the selected predictor variables using mapIV.
  • _Depending on the returntype, it either returns a summary table, individual IV tables, plots, or a combination of these.

3. Function Relationships:

• _createIV calls:
  • _ptest to filter the dataset based on the significance level.
  • _mapIV to calculate IV values for selected predictor variables.
  • _Depending on the return_type, it may call plot_WOE or create_barasis to visualize the results.

4. Overall Description:

• The code aims to analyze the predictive power of predictor variables using IV and related statistics.
• It provides functions to calculate IV, perform statistical tests, and visualize the results.
• _The main function createIV serves as a controller function that manages the entire process from data filtering to visualization, offering flexibility in terms of the type of output desired.

[martinctc]

Great work @sachinstl - thank you for your contributions! This is now available on PyPI as v0.3.0. Merging this in now.