A2 Task 1: Design a big data analytics project by following Big Data Analytics Lifecycle.

Big Data Lifecycle

Discovery

Learn Business Domain: Understand the industry and context. Interview Sponsor & Identify Stakeholders: Engage with key individuals to gather insights. Define Resources & Goals: Establish objectives and available resources. Identify Potential Data Sources: Locate relevant data sources. Frame the Problem & Develop Initial Hypotheses: Formulate hypotheses, including Null Hypothesis (H0) and Alternative Hypothesis (HA or H1).

Data Preparation

Prepare Sandbox: Set up an environment for data preparation. Perform ETLT (Extract, Transform, Load, Transform): Process data for analysis. Understand Data Details: Examine the data’s structure and quality. Data Conditioning: Address issues like missing values and outliers. Format Data: Prepare data for analysis. Visualize Data: Use plots to explore data patterns.

Model Planning

Select Variables: Based on relationships (e.g., correlation matrix) and domain knowledge. Identify Candidate Models: Refer to hypotheses, translate into machine learning models, review literature, and document assumptions.

Model Building

Create Datasets: Prepare training, validation, and testing datasets. Train and Test Models: Evaluate model performance.

Communicating Results

Compare Results: Assess against criteria. Articulate Findings: Clearly present results. Discuss Limitations & Recommendations: Provide insights on limitations and suggest improvements.

Operationalize

Deliverables: Finalize and deliver the project. Pilot Project: Test the model in a real-world scenario. Performance & Constraints: Monitor and address any constraints. Training: Educate new users as needed.

1. Learn Business Domain

Objective: The project aims to identify profiles on social networks that are mistakenly recorded as human or non-human profiles (e.g., bots).
Specific Task: You need to identify if the user’s gender is either male, female, or a brand based on their Twitter profile data and associated tweets.
Dataset: The dataset for this classification task can be accessed via Kaggle at the Twitter User Gender Classification dataset.

2. Define Resources & Goals

Data Source: Twitter user profiles including tweets, descriptions, link color, sidebar color, and other metadata.
Primary Task: Classify gender into three categories: male, female, or brand.
Key Questions:
- How well do the words in tweets and profiles predict the user’s gender?
- What are the specific words that strongly predict male or female gender?
- How well do stylistic factors (like link color and sidebar color) predict gender?

3. Frame the Problem & Develop Initial Hypotheses

Problem Type: This is a supervised learning problem where the task is to predict a categorical variable (gender: male, female, or brand).
Hypotheses:
- Null Hypothesis (H0): Words in tweets and profiles do not have a significant effect on predicting user gender (i.e., the predictive power is random or weak).
- Alternative Hypothesis (H1): Words in tweets and profiles have a significant effect on predicting user gender (i.e., they provide strong predictive power).
Additional Hypotheses:
- H0: Stylistic factors (link color, sidebar color) are not good predictors of gender.
- H1: Stylistic factors can strongly predict gender.

4. Data Preparation - @kgoel59

Preprocessing Steps:
- Stemming & Lemmatization: To reduce words to their base or root forms.
- Lowercasing: To eliminate discrepancies between uppercase and lowercase letters.
- Tokenization: Split text into individual words or tokens.
- Embedding: Use word embeddings (e.g., Word2Vec, GloVe, or TF-IDF) to represent text data numerically.
- Bag of Words: A simple and effective method for text representation.

5. Visualize Data

Visualization Techniques:
- Word Cloud: To display frequent words by gender category (male/female/brand).
- Word Distance/Similarity: Use techniques like cosine similarity to understand which words or phrases are closer to each gender category.

6. Model Building

Classification Models:
- K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Decision Trees, Random Forest – led by @banin-sensha.
Clustering Models:
- K-Means, DBSCAN, Self-Organizing Maps (SOM) – led by @alvnj and @Rishab2k .
Regression Models:
- Linear Regression, Logistic Regression – led by Lino.
Neural Networks:
- Build deep learning models to classify gender using Neural Networks – led by @Ashb8 .

7. Training and Testing

Cross-Validation: Use k-fold cross-validation to prevent overfitting and assess model generalization.
Grid Search: Optimize hyperparameters by performing grid search to select the best model configuration.

8. Deliverables

Evaluation Metrics:
- Confusion Matrix: To visualize the performance of each classification model.
- Accuracy: The percentage of correctly classified instances.
- Precision: The proportion of true positives out of all positive predictions.
- Recall: The proportion of true positives out of all actual positives.
- Loss Curves: For neural networks, to visualize the training process and detect overfitting or underfitting.
Reporting:
- Each team member will write their individual reports in LaTeX, and the reports should include an analysis based on the hypotheses at each stage (preprocessing, model building, evaluation).

kgoel59 / BigData