highfestiva / finplot

Performant and effortless finance plotting for Python
MIT License
920 stars 187 forks source link

[BUG] PerformanceWarning DataFrame is highly fragmented. #521

Closed IlanKalendarov closed 5 months ago

IlanKalendarov commented 5 months ago

Requirements (place an x in each of the [ ])**

Code to reproduce

import finplot as fplt
import pandas as pd

def create_chart(self):
        self.add_hh_ll_hl_lh_labels()
        self.add_buy_entries()
        self.add_trade_boxes()
        self.add_trend()
        fplt.lod_labels = 200
        self.add_news_labels()
        if self.realtime:
            fplt.timer_callback(self.update_data, 3.0)
        fplt.show()

def update_data(self):
        """
        Updates the data by reading a CSV file, converting the date column to datetime,
        resetting the index, and renaming the index column.
        """
        script_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
        parent_dir = os.path.dirname(script_dir)
        data_file_path = os.path.join(parent_dir, 'BackEnd', 'data.csv')
        with open(data_file_path, 'r') as f:
            self.df = pd.read_csv(f)
        self.df['date'] = pd.to_datetime(self.df['date'], errors='coerce')
        self.df['date'] = self.df['date'].astype('int64')
        self.df.reset_index(inplace=True)
        self.df.rename(columns={'index': 'Unnamed: 0'}, inplace=True)
        self.plot_candles.candlestick_ochl(
            self.df[['date', 'open', 'close', 'high', 'low']])
        self.add_hh_ll_hl_lh_labels()
        self.add_trend()
        self.add_news_labels()
        self.add_buy_entries()
        self.add_trade_boxes()

Describe the bug

When trying to use the realtime method what I do is that i have in the background a process that updates a csv file every x seconds. And using finplot I am reading the csv file using the timer_callback method which will use the update_data method.

after some time i get this error:

/opt/homebrew/lib/python3.11/site-packages/finplot/init.py:442: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use newframe = frame.copy() output_df[col] = df[col]

Is there a better way to do it without any performance sacrifice ?

Reproducible in:

OS: Mac OS finplot version: 1.9.5 pyqtgraph version: 0.13.3 pyqt version: PyQt6

highfestiva commented 5 months ago

I've never seen this, it's a warning from Pandas. My guess is that this isn't caused by finplot, but by your code (possibly by adding new columns to the data you update). You've not supplied the data and complete code to run a minimal example.