shankariraja / StockSence

MIT License

0 stars 0 forks source link

Tracking Issue: StockSense #1

Open shivas1516 opened 2 months ago

shivas1516 commented 2 months ago

Project Tracking Issue: StockSense

Project Overview

Developing an automated stock price prediction system using big data technologies and machine learning.

Progress Checklist

Phase 1: Setup and Data Extraction

[x] Set up development environment
[x] Implement data extraction from Yahoo Finance
[x] Configure Kafka
[x] Create batch data pipeline

Phase 2: Data Processing and Storage

[x] Develop PySpark session for data processing
[ ] Implement data transformation and feature engineering
[ ] Set up PostgreSQL
[ ] Implement data loading to PostgreSQL

Phase 3: Machine Learning Model Development

[ ] Explore and analyze historical data using PySpark
[ ] Preprocess data for machine learning
[ ] Develop and train the initial Stacked LSTM model
[ ] Implement model evaluation (RMSE, MAE)
[ ] Implement model selection process

Phase 4: Pipeline Orchestration

[ ] Set up Apache Airflow
[ ] Develop DAG for weekly pipeline execution
[ ] Implement error handling and logging

Additional Tasks

[ ] Develop a user-friendly dashboard for predictions and visualizations
[ ] Conduct thorough system testing
[ ] Optimize the performance of each component
[ ] Prepare comprehensive project documentation

Project Proposal Link

Gdoc: https://docs.google.com/document/d/1hPaFcnJrjHFJDNDNgop1AohrnhG3Pf2j6A65vkRXzw4/edit

Please update this issue as progress is made. Use comments for discussions or to report major milestones.

shankariraja commented 2 months ago

@shivas1516 Thanks for creating the tracking issue