Implement new explore_num() method: 'multicollinearity'

Description:

Method Functionality Idea:

The multicollinearity method checks for multicollinearity among numerical variables using Variance Inflation Factors (VIF). Multicollinearity occurs when independent variables in a regression model are highly correlated with each other, leading to issues with parameter estimation and interpretation.

How it operates:

The method first prepares the data by removing missing values. Then, it calculates the VIF for each numerical variable using the calculate_vif function. The VIF measures how much the variance of an estimated regression coefficient is inflated due to multicollinearity. Higher VIF values indicate stronger multicollinearity concerns. The method returns a DataFrame displaying the VIF values for each numerical variable.

Usage:

To check for multicollinearity among numerical variables using VIF:

vif_df = explore_num(df, numerical_variables, method='multicollinearity')
print(vif_df)

This method returns a DataFrame containing the VIF values for each numerical variable.

Example:

# Check for multicollinearity using VIF for numerical variables 'Feature1' and 'Feature2'
vif_df = explore_num(df, ['Feature1', 'Feature2'], method='multicollinearity')
print(vif_df)

Notes:

Multicollinearity can lead to issues such as unstable parameter estimates and difficulties in interpreting regression coefficients.
A VIF value greater than 10 is commonly used as a threshold to indicate potential multicollinearity concerns.

Implementation Summary:

The implementation of the multicollinearity method within the explore_num() function is designed to check for multicollinearity among the numerical variables in a DataFrame. This is done using the Variance Inflation Factor (VIF). Multicollinearity can impact the stability and interpretability of regression models, and this method helps to identify such issues.

Code Breakdown:

Function Definition and Parameters:

def explore_num(
        df: pd.DataFrame,
        numerical_variables: List[str],
        method: str = 'all',
        output: str = 'print',
        threshold_z: int = 3
) -> Optional[str]:
    """
    Analyze numerical variables in a DataFrame for distribution characteristics, outlier detection, normality tests, skewness, kurtosis, correlation analysis, and multicollinearity detection.

    # Full docstring continues...
    """

Purpose: The explore_num() function analyzes numerical variables in a DataFrame.
Parameters: The function takes a DataFrame (df), a list of numerical variables (numerical_variables), a method (method), an output format (output), and a threshold for Z-score outliers (threshold_z).

Method: 'multicollinearity'

if method.lower() in ['multicollinearity', 'all']:
    # use non-na df: data
    data = df.copy()
    data = data[numerical_variables].dropna()

    vifs = calculate_vif(data, numerical_variables)
    result.append(f"\n<<______MULTICOLLINEARITY CHECK - VIF______>>\n")
    result.append(f"Variance Inflation Factors:\n{vifs.to_string()}\n")
    result.append("☻ Tip: VIF > 10 indicates potential multicollinearity concerns.")

Purpose: This section checks for multicollinearity using VIF.
Steps:
- Copy DataFrame: A copy of the DataFrame is created and non-NA numerical variables are retained.
- Calculate VIF: The function calculate_vif() is used to calculate the Variance Inflation Factor for the numerical variables.
- Append Results: The results are appended to the output list (result), with a tip indicating potential concerns if the VIF exceeds 10.

Combining Results and Output:

combined_result = "\n".join(result)

if output.lower() == 'print':
    print(combined_result)

    if method.lower() == 'multicollinearity':
        return vifs
elif output.lower() == 'return':
    if method.lower() in ['all', 'multicollinearity']:
        return combined_result

    if method.lower() == 'multicollinearity':
        return vifs

Purpose: This section combines the results and determines the output format.
Output: Depending on the output parameter, the results are either printed or returned. For the multicollinearity method, the VIF DataFrame is returned when output is set to 'return'.

See the Full Function:

The full implementation can be found in the datasafari repository.

ETA444 / datasafari