search

ed-pilots-network / backend

Project to consume the Kafka, process the messages and provide an API to access the data
Apache License 2.0
12 stars 5 forks source link

Enhancing Performance in Trade Module with Improved Market Data Management #140

Closed pveeckhout closed 9 months ago

pveeckhout commented 9 months ago

Is your feature request related to a problem? Please describe. The current setup of the trade module utilizes a view on the 'historic market data' for performing trade route calculations. However, this approach is proving to be inefficient and not performant enough. The delay and lag in processing this data are causing significant challenges in timely route analysis, which is critical for our operations.

Describe the solution you'd like I propose a restructuring of how we handle market data. Instead of relying solely on a view of historic market data, we should implement a system where market data is inserted directly as it arrives. This change would involve maintaining two types of data: historic and current.

  1. Historic Data: All incoming market data should be saved to the historic data table. This ensures we have a comprehensive and continuous record of market trends over time.
  2. Current Data: The current data table should only be updated with new incoming data if the data's timestamps are more recent than what is currently stored. This approach ensures that the current data table always reflects the most up-to-date market information.

By separating the handling of historic and current data, we can optimize performance, especially in trade route calculations, as the current data table will be more streamlined and faster to query.

Describe alternatives you've considered An alternative could be to optimize the existing view on the historic market data. However, this might not offer the same performance improvements as the proposed solution and could still result in delays due to the inherent complexity of continuously updating a view with large volumes of data.

Additional context This change is crucial for improving the efficiency of our trade route calculations. The current performance issues are a bottleneck in our operations. Implementing this feature will significantly enhance our module's responsiveness. Additionally, the proposed solution aligns well with our long-term goal of building a robust and scalable trade analytics platform.

pveeckhout commented 9 months ago

blocks https://github.com/ed-pilots-network/backend/issues/51

pveeckhout commented 9 months ago

already implemented