nautechsystems / nautilus_trader

A high-performance algorithmic trading platform and event-driven backtester
https://nautilustrader.io
GNU Lesser General Public License v3.0
2.18k stars 498 forks source link

Improve StreamingFeatherWriter to support file rotation and better time management #1945

Closed graceyangfan closed 1 month ago

graceyangfan commented 2 months ago

Issue Description

The current StreamingFeatherWriter class lacks file rotation functionality, which may lead to oversized files during long-running operations or high data volume scenarios. Additionally, there's room for improvement in time management and file naming.

Suggested Improvements

  1. Implement file rotation functionality, supporting rotation based on size, time interval, and scheduled dates.
  2. Use a Clock object for time management to improve flexibility and testability.
  3. Add timestamps to filenames for better organization and identification.
  4. Track file sizes and creation times to support rotation functionality and provide more information.
  5. Add methods to query current file information and next rotation time.

Expected Benefits

Related Files

cjdsellers commented 1 month ago

Implemented in #1954, many thanks @graceyangfan!