Time Series Analysis: Understanding Melbourne Housing Trends

Introduction to Time Series Analysis with Cleaned Melbourne Housing Data

Issue Description

This issue tracks the journey of diving into time series analysis on the Melbourne housing market, utilizing the cleaned dataset (1_clean_melb_data.py). The primary focus is to grasp the fundamentals of time series modelling and to observe housing market trends and seasonality.

Strategy and Learning Approach

Data Utilization: The analysis will start with the dataset from 1_clean_melb_data.py, which is cleaned but not transformed, to retain the original scale of price data.
Outliers in Land Size:
- Initially, outliers in land size are removed to simplify the modelling process. This step is intended to facilitate a smoother entry into time series analysis for learning purposes.
- As understanding deepens, the plan is to reintroduce these outliers. This will help assess their impact on the model and provide comparative insights.
Learning Focus: The primary aim is to build a strong understanding of time series concepts and methodologies as applied to housing market data.

Documentation and Evolution of Analysis

Progress Updates: Regular updates will be posted, detailing the learning process, any challenges faced, and insights gained throughout the analysis.
Adaptive Strategy: Depending on the initial findings and learning curve, the strategy might evolve to incorporate more complex aspects, including analyzing the effect of outliers.
Comparative Exploration: Once comfortable with the basic time series models, a comparison will be made by including land size outliers, aiming to explore and understand their specific impact on the analysis.

Goal

The overarching goal is to develop a solid foundation in time series analysis applied to real-world data, starting with a more straightforward dataset and gradually introducing complexities like outliers to deepen market trend understanding.

Update on Time Series Analysis: Progress and Revised Approach

Progress Overview:

Initial Steps with Moving Average: I started by attempting to implement a moving average to observe long-term trends in housing prices. This process led to a realization that the structure of my dataset was not optimized for time series modelling.
Data Restructuring and Preparation: I identified that the essential elements for my analysis are the date and price. Therefore, I reformatted the data, ensuring that the 'Date' column is correctly parsed as a datetime object. This step is crucial as many time series functions heavily rely on date information.
Simplification of Dataset: I opted to create a new DataFrame (date_avgprice), focusing solely on 'Date' and 'Price'. This DataFrame aggregates the data by date, calculating the average price for each day. This approach effectively handles multiple entries per date and simplifies the dataset for clearer analysis.

Revised Strategy and Next Steps:

Data Preparation: The dataset is now appropriately structured with 'Date' as the index and 'Price' as the main variable of interest.
Implementing Moving Average and Exponential Smoothing: My immediate next steps involve applying a moving average and exponential smoothing to the data. This will help in understanding both short-term and long-term price trends.
Advanced Time Series Models: Post the initial analysis, I plan to explore ARIMA or Seasonal ARIMA models for a more nuanced understanding and forecasting of market trends.
Decomposition and Integration with Regression Findings: Following the advanced modelling, I will engage in decomposition to dissect the trends further. This will also be an opportunity to reflect on my regression findings and interpret the patterns in relation to time-based trends.
Visualization and Interpretation: The final steps will involve visualizing these trends and interpreting the patterns, providing a comprehensive view of the Melbourne housing market's dynamics over time.

Key Considerations:

The primary focus of the analysis remains on understanding market-level trends.
The insights gained from time series analysis are separate yet complementary to the regression analysis findings.
The simplified dataset and structured approach aim to provide a clear perspective on how the housing market has evolved over time.

Conclusion:

This updated approach and guide will help in gaining a deeper understanding of the Melbourne housing market, balancing the focus on broader market trends through time series analysis and detailed insights from the regression analysis. The goal is to achieve a holistic view of the market dynamics, combining temporal patterns with property characteristics.

Update on Time Series Analysis: Moving Average Insights and Transition to Exponential Smoothing

Recent Accomplishments:

Successful Implementation of Moving Average: Utilized Pandas to compute the moving average for the Melbourne housing data, smoothing out short-term fluctuations to reveal long-term trends.
Insightful Visual Analysis: Produced a line plot placing the average house price with the rolling average and the number of houses sold on the same graph. This visual representation facilitated a multi-faceted analysis of the data.
Enhanced Graph Interpretation: Improved the graph's clarity by adjusting the transparency (alpha) and refining the legend, allowing for immediate and detailed visual insights.

Detailed Analysis of Moving Average Results:

The analysis began with an initial observation of the trends and volumes, noting the correlation between the number of sales and average prices.
A deeper dive with research added context to the trends, linking them to economic indicators and government interventions during the period between 2016 and 2017.
Further exploration shed light on legislative impacts and market self-correction mechanisms, with a keen focus on the interplay between foreign investment and regulatory changes.

Analysis Summary:

The moving average graph, accompanied by the volume of sales, revealed both expected and unexpected market behaviours.
Detailed analysis, provided in three stages, allowing for a granular understanding of market dynamics, legislative impacts, and potential seasonality effects.

Detailed Analysis and Graphical Context:

ma graph

First Analysis (Initial Observations):

Observation of Trends: Noticed an initial steady price which then increased with the number of houses sold.
Volume and Price Correlation: As sales volume increased, so did the average price, suggesting market demand impacts prices.
Market Dynamics: Identified a dip in both prices and the volume of sales at one point, speculating a cooling-off period or a reaction to high prices.
Rolling Average Analysis: The 7-day rolling average began to show an upward trend, suggesting a general increase in housing prices over time.
Market Correction Speculation: Noticed a sharp decrease in the rolling average, initially thought to be a crash, but later considered a market correction due to overheating.
Post-Correction Recovery: Prices and sales volume picked up again after the correction, with speculation on the buyers' psychology during this period.
Seasonality Speculation: Suggested that certain times of the year showed decreased activity and prices, questioning if this was due to seasonal effects.

Second Analysis (In-depth with Research):

Market Correction Explanation: Clarified that the perceived market crash was actually a correction due to an overheated market.
Economic Growth Influence: Linked real estate demand and price increases to Melbourne's economic growth, low unemployment rates, and overall real estate trends between 2016 and 2017.
Rolling Average Trend: Reiterated that the rolling average showed an upward trend due to strong market demand and limited housing supply.
Foreign Investment Impact: Discussed the influence of foreign investment on the housing market, leading to higher prices.
Government Intervention: Speculated that government policies to restrict foreign investment cooled the market and led to the observed correction.
Seasonal Patterns Elaborated: Provided specific months for seasonal patterns observed in the Melbourne real estate market:
- Summer (December - February): The market slows down initially but picks up towards the end.
- Autumn (March - May): Considered a favourable time for sales, typically sees increased activity.
- Winter (June - August): Typically experiences a slowdown.
- Spring (September - November): Busiest season with a surge in listings and sales.

Third Analysis (Further Explanation and Speculation):

Market Self-Correction: Suggested that the market correction was a multifaceted event, not solely due to seasonality, but significantly influenced by government awareness and intervention.
Legislation on Underquoting: Identified the passage of legislation aimed at underquoting as a key government measure. This was a direct response to the overheated market caused by the discrepancy between estimated and actual property prices, which attracted both residents and investors, leading to escalated housing prices.
OTOP Property Investment: Noted legislation made to make off-the-plan (OTOP) property investments less attractive to investors, which was one of the methods used to temper the influx of investors and redirect the focus towards actual residents.
Increased Taxation on Foreign Investors: Highlighted additional legislative measures to cool down the market by imposing a greater tax burden on foreign investors under certain conditions.
Continued Monitoring: Acknowledged that the data ends in October 2017, necessitating further evaluation using other time series models to forecast beyond this point.
Data Limitation Acknowledgment: Recognized that the analysis is limited by the dataset's timeline, which ends in October 2017.

Next Steps in the Time Series Journey:

Shifting to Exponential Smoothing: With valuable insights obtained from the moving average model, I am now preparing to explore the Exponential Smoothing model. The choice to explore Exponential Smoothing next is driven by intriguing patterns observed towards the end of the dataset. While the market demonstrated a booming phase, it eventually cooled off, almost suggesting a crash, before stabilizing. Given that Exponential Smoothing gives more weight to recent observations, it seems particularly well-suited to analyze this phase and provide additional insights into the recent market behaviour.
Anticipated Insights: By applying Exponential Smoothing, I aim to dissect the cooling-off period more closely and understand the stabilization that follows. This model's sensitivity to recent changes could shed light on whether the cooling-off signifies a temporary dip or a more substantial shift in the market.
File Organization for Model Segregation: As I transition from one model to another, I will maintain organized code by creating separate files for each. The current file 4_time_series_model.py will be refactored to 4.1_moving_average.py.
Upcoming File Creation: The next file in sequence will be 4.2_exponential_smoothing.py, which will be dedicated to implementing and analyzing the Exponential Smoothing model.

Reflection:

The Moving Average model provided a solid foundation for understanding the Melbourne housing market's trends. The analyses have set the stage for more advanced time series modelling techniques.
A systematic approach to documenting and separating the code for each model will ensure that the project remains well-organized and that findings are clearly communicated.

Update on Transitioning to Exponential Smoothing for Time Series Analysis

Refining the Data Frame for Time Series Analysis:

Adapting from Previous Work: After reviewing the code from 4.1_moving_average.py, I tailored the DataFrame specifically for time series analysis, focusing solely on price data and removing the count aggregation from the previous code.

Tackling Inconsistent Date Frequencies:

Issue of Date Inconsistencies: Encountered a hurdle with date inconsistencies in our data—some dates were a week apart, others a month, disrupting the time series flow.
Utilizing Pandas Resample Function: To overcome this, I employed the resample function in Pandas, setting a uniform frequency and aggregating data using the mean.
Initial Scarcity of Data Points: This approach initially led to a scarcity of data points due to the timeline constraints of the dataset.

Navigating Through Resampling Frequencies:

Monthly vs. Weekly vs. Bi-Weekly Resampling: Monthly resampling proved too scarce. Switching to weekly showed promise but resulted in numerous missing entries. After trying monthly and weekly, a bi-weekly frequency emerged as the optimal choice, offering a balance with fewer missing data points.
Specific Missing Data Issue: Notably, the resampled dataset was missing entries due to large gaps between dates in the original dataset, which the resampling function couldn't bridge.

Resolving Missing Data Entries:

Rejected Approaches: Dismissed several approaches for handling missing data, like dropping missing dates (which would reintroduce inconsistencies) or filling with zeros (which was illogical for our dataset).
Chosen Method - Forward Filling: Eventually, I chose to forward-fill the missing entries, ensuring a continuous, consistent timeline crucial for accurate time series analysis. This method assumes that days without data can be estimated using the last available data point, which is logical given the nature of housing market trends.

Preparation for Exponential Smoothing:

Importance of Data Order in Splitting: In the process of splitting the data for training and testing, I noted the unique consideration for time series where the order of data is paramount, contrasting with the random sampling acceptable in regression analysis.
Choosing a Data Split Ratio: With a limited dataset, an 80-20 split was chosen, balancing the need for sufficient training data against the requirement to validate the model's predictive power.

Implementing the Holt-Winters Seasonal Model:

Importing Necessary Library for Exponential Smoothing: Initiated the process by importing ExponentialSmoothing from statsmodels.tsa.holtwinters. This library is crucial for applying the Holt-Winters method to our time series data.
Model Selection Based on Previous Observations: With the data prepared and the necessary library imported, I then initialized the Holt-Winters seasonal model. This model was chosen based on previous observations from the moving average analysis, which indicated both a trend and seasonality in the data. These patterns are often linked to Melbourne's weather seasons and the associated housing market activity.

Model Optimization Challenges:

Convergence Issues with the Model: A snag appeared when the model failed to converge, indicating trouble with the automatic optimization of the alpha, beta, and gamma parameters.
Need for Hands-On Parameter Tuning: This was a clear sign that a more hands-on approach to parameter tuning was necessary.

Forthcoming Strategy to Optimize Parameters:

Utilizing Grid Search and Cross-Validation: My next step involves using grid search combined with cross-validation to manually find the optimal parameters.
Alternative Adjustments if Needed: If this method doesn't yield results, I will adjust the model's arguments, potentially employing a Box-Cox transformation or adjusting for bias, to coax the model into convergence.

Optimizing Holt-Winters Exponential Smoothing Parameters

Implementing Grid Search for Parameter Optimization:

Grid Search Setup: Configured a grid search to explore various combinations of parameters for the Holt-Winters model. The search grid included options for trend and seasonality (additive, multiplicative, or none) and a range of smoothing levels for the level, trend, and seasonality components of the model.

Cross-Validation Approach:

Time Series Split: Employed TimeSeriesSplit from Scikit-learn for cross-validation, acknowledging the sequential nature of time series data where random sampling isn't appropriate.

Parameter Grid Details:

Seasonal Periods Correction: Realized that the seasonal periods should be six, not eight, based on Melbourne's four weather seasons affecting the real estate market. This revelation came after recalculating that each season lasts roughly three months, equating to six bi-weekly periods.

Execution of Grid Search:

Iterative Process: Executed an exhaustive search over the defined parameter grid, recording mean squared error (MSE) scores for each parameter combination while handling exceptions for invalid parameter sets.

Encountered Issues and Insights:

Unexpected Optimal Parameters: The grid search concluded that the best parameters were with 'trend' and 'seasonal' set to 'None', suggesting that a simple exponential smoothing model without trend or seasonality might be more appropriate for the data.
Reflection on Results: I'm puzzled by these results because our moving average analysis indicated seasonality. This outcome could hint that the resampling might be averaging out the trends and seasonal fluctuations, leading to the model's recommendation against using a trend or seasonal components.

Next Steps:

Exploring Simple Exponential Smoothing: Plan to test a simple exponential smoothing model next to see if it aligns better with the identified parameters.
Considering ARIMA Models: If the simple approach doesn't capture the data's nuances, I might explore ARIMA or Seasonal ARIMA models for potentially better time series modelling.
Investigating Underlying Causes: Need to delve deeper into why the grid search is steering away from a trended and seasonal model. This may involve revisiting the resampling approach or the data itself for clues.

Conclusion and Forward Planning:

Model Selection Dilemma: The grid search's recommendation to disregard trends and seasonality raises questions about the best path forward.
Ensuring Model Fit: It's crucial to ensure that the chosen model accurately reflects the underlying data patterns, especially given the previous indications of trend and seasonality.
Adjusting Strategy as Needed: Ready to adjust my strategy based on further testing and insights, with a focus on selecting the most fitting time series model for the Melbourne housing market data.

Conclusion of Time Series Analysis Section

Realizations and Decisions:

After thorough investigation and attempts at traditional time series modelling, I've concluded that this approach is not viable for the Melbourne real estate data due to the inherent inconsistencies in date entries.
Resampling, which initially seemed like a potential solution, unfortunately, averages out the crucial elements of seasonality and trends that are significant in our dataset.

Insights from Moving Average and Exponential Smoothing:

The Moving Average analysis confirmed the presence of trends and seasonal patterns, which was further supported by cross-referencing with historical events and regulations in Melbourne during the time periods in question.
However, when attempting to fine-tune the Exponential Smoothing model with grid search and cross-validation, the results indicated that our data does not exhibit the necessary characteristics (trend and seasonality) for this model to be effective.

Implications for Time Series Modelling:

The data's date inconsistencies make it incompatible with traditional time series models like ARIMA and Seasonal ARIMA.
Given that resampling distorts the data's inherent seasonality and trend, it's not a feasible method for preparing our dataset for these models.

Moving Forward:

This brings the traditional time series analysis portion of the project to a close. The evidence suggests that the best way to analyze our data moving forward is through machine learning approaches rather than traditional time series models.
This project intended to initially apply traditional data analysis techniques (cleaning, EDA, regression, and time series) before delving into machine learning methods to compare the outcomes. However, we will bypass the traditional time series analysis due to the data constraints.

Next Steps:

I will be closing this branch of the project and transitioning to the final phase: creating an informative and interactive dashboard that encapsulates all the findings and insights gathered so far.
The focus will now shift to presenting the data insights in a comprehensive dashboard format.

Reflection:

While it's disappointing that traditional time series analysis is not applicable in this instance, the journey has provided valuable learning experiences about the nature of time series data and the importance of data consistency.

arahm071 / Melbourne-Modeling-Journey