Automated Data Processing and Analysis

[sarthakforwet]

@DhanushNehru, I would like to add the following script -

A Python Script to perform data analysis and processing for a Regression task given a dataset and the target variable as input.

[TheProgrammer-creator]

import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error, r2_score import matplotlib.pyplot as plt import seaborn as sns

1. Load the dataset

def load_data(file_path): data = pd.read_csv(file_path) print("Data successfully loaded.") return data

2. Exploratory Data Analysis

def exploratory_data_analysis(data, target_variable): print("First 5 rows:\n", data.head()) print("Data information:\n", data.info()) print("Descriptive statistics:\n", data.describe())

# Check correlation
plt.figure(figsize=(10, 8))
sns.heatmap(data.corr(), annot=True, cmap="coolwarm")
plt.title("Correlation Matrix")
plt.show()

# Target variable distribution
sns.histplot(data[target_variable], kde=True)
plt.title(f"Distribution of Target Variable: {target_variable}")
plt.show()

3. Data Preprocessing

def preprocess_data(data, target_variable):

Separate features and target variable

X = data.drop(columns=[target_variable])
y = data[target_variable]

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

print("Data preprocessing complete.")
return X_train, X_test, y_train, y_test

4. Train the Model

def train_model(X_train, y_train): model = LinearRegression() model.fit(X_train, y_train) print("Model trained successfully.") return model

5. Evaluate the Model

def evaluate_model(model, X_test, y_test): y_pred = model.predict(X_test) mse = mean_squared_error(y_test, y_pred) r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error (MSE): {mse}")
print(f"R^2 Score: {r2}")

# Visualize Predictions
plt.scatter(y_test, y_pred, alpha=0.7)
plt.xlabel("Actual Values")
plt.ylabel("Predictions")
plt.title("Actual Values vs. Predictions")
plt.show()

Main function to run the entire script

def main(file_path, target_variable): data = load_data(file_path) exploratory_data_analysis(data, target_variable) X_train, X_test, y_train, y_test = preprocess_data(data, target_variable) model = train_model(X_train, y_train) evaluate_model(model, X_test, y_test)

Execute script by specifying the file path and target variable

file_path = "path_to_your_dataset.csv" target_variable = "your_target_variable_name" main(file_path, target_variable) Explications: Load the dataset: Reads data from a CSV file. Exploratory Data Analysis (EDA): Displays initial data, statistics, and correlation matrix for understanding relationships. Data Preprocessing: Splits the dataset into training and testing sets and applies standardization. Train the Model: Trains a linear regression model on the training data. Evaluate the Model: Calculates Mean Squared Error and R^2 Score and plots predictions vs. actual values for model performance.