Not all timeframes are rendering chart data

[jlixfeld]

When cycling through my list of timeframes, I don't get charts rendered for all of them, only 1m, 5m, d, w, m, q, y. 15m, 30m, 1h, 4h, 6h, 12h all show the last price, but no chart data:

https://github.com/louisnw01/lightweight-charts-python/assets/17339017/7c0d7ea2-1182-47b6-bddf-1d55160252e4

from lightweight_charts import Chart
from pandas import DataFrame
from modules import settings
from modules.candles import Candles
import asyncio
import modules.database as db

database = db.Database(url="postgresql://postgres@127.0.0.1:5432/ariobot")

async def get_bar_data(symbol, timeframe):
    contract = database.Contracts.get(symbol=symbol)
    dataframe = DataFrame(Candles(database, contract, timeframe, 'chart').get())
    print(symbol, timeframe, len(dataframe), dataframe.info())
    return dataframe

class API:
    def __init__(self):
        self.chart = None

    async def on_search(self, symbol):
        timeframe = self.chart.topbar['timeframe'].value
        dataframe = await get_bar_data(symbol, timeframe)
        if not dataframe:
            return
        self.chart.set(dataframe)
        self.chart.topbar['symbol'].set(symbol)

    async def on_timeframe(self):
        timeframe = self.chart.topbar['timeframe'].value
        symbol = self.chart.topbar['symbol'].value
        dataframe = self.chart.set(await get_bar_data(symbol, timeframe))
        if not dataframe:
            return
        self.chart.set(dataframe)

async def main():
    api = API()

    chart = Chart(api=api, volume_enabled=False, topbar=True, searchbox=True, toolbox=True, debug=True)
    chart.legend(True)
    chart.precision(6)

    symbols = [contract.symbol for contract in database.Contracts.select()]
    timeframes = list(settings.Settings().TIMEFRAMES)

    chart.topbar.textbox('symbol', symbols[0])
    chart.topbar.switcher('timeframe', api.on_timeframe, *timeframes, default=timeframes[0])

    contract = database.Contracts.select().where(database.Contracts.symbol == symbols[0]).get()
    chart.set(DataFrame(Candles(database, contract, timeframes[0], 'chart').get()))

    await chart.show_async(block=True)

if __name__ == '__main__':
    asyncio.run(main())

The prints in the get_bar_data() function don't indicate any obvious difference in the data between working (5m, d) timeframes and one that doesn't work (15m).

# frontend2.py
[pywebview] Using Cocoa
<class 'pandas.core.frame.DataFrame'>
Index: 3221 entries, 0 to 1569362
Data columns (total 5 columns):
 #   Column  Non-Null Count  Dtype              
---  ------  --------------  -----              
 0   time    3221 non-null   datetime64[ns, UTC]
 1   open    3221 non-null   float64            
 2   high    3221 non-null   float64            
 3   low     3221 non-null   float64            
 4   close   3221 non-null   float64            
dtypes: datetime64[ns, UTC](1), float64(4)
memory usage: 151.0 KB
USDCHF 5m 3221 None
<class 'pandas.core.frame.DataFrame'>
Index: 2245 entries, 0 to 523120
Data columns (total 5 columns):
 #   Column  Non-Null Count  Dtype              
---  ------  --------------  -----              
 0   time    2245 non-null   datetime64[ns, UTC]
 1   open    2245 non-null   float64            
 2   high    2245 non-null   float64            
 3   low     2245 non-null   float64            
 4   close   2245 non-null   float64            
dtypes: datetime64[ns, UTC](1), float64(4)
memory usage: 105.2 KB
USDCHF 15m 2245 None
<class 'pandas.core.frame.DataFrame'>
Index: 1417 entries, 0 to 5449
Data columns (total 5 columns):
 #   Column  Non-Null Count  Dtype              
---  ------  --------------  -----              
 0   time    1417 non-null   datetime64[ns, UTC]
 1   open    1417 non-null   float64            
 2   high    1417 non-null   float64            
 3   low     1417 non-null   float64            
 4   close   1417 non-null   float64            
dtypes: datetime64[ns, UTC](1), float64(4)
memory usage: 66.4 KB
USDCHF d 1417 None

Any clues?

[louisnw01]

Hey,

After you convert your database call to a pd.DataFrame (line 12) try adding the following lines:

dataframe.dropna(inplace=True)
dataframe.drop_duplicates(subset='time', inplace=True)
dataframe.reset_index(inplace=True)

If this works, you may only need one or two of these lines to clean the data.

[jlixfeld]

No change.

[jlixfeld]

For completeness, here's the code as it stands (I've made some other changes in the mean time):

from lightweight_charts import Chart
from pandas import DataFrame, Series
from modules import settings
from modules.candles import Candles
import asyncio
import modules.database as db

database = db.Database(url="postgresql://postgres@127.0.0.1:5432/ariobot")

async def get_bar_data(symbol, timeframe, chart_type):
    contract = database.Contracts.get(symbol=symbol)
    dataframe = DataFrame(Candles(database, contract, timeframe, chart_type).get())
    dataframe.dropna(inplace=True)
    dataframe.drop_duplicates(subset='time', inplace=True)
    dataframe.reset_index(inplace=True)
    return dataframe

class API:
    def __init__(self):
        self.chart = None

    async def on_search(self, symbol):
        timeframe = self.chart.topbar['timeframe'].value
        dataframe = await get_bar_data(symbol, timeframe, 'chart')
        if dataframe.empty:
            return
        self.chart.set(dataframe)
        self.chart.topbar['symbol'].set(symbol)

    async def on_timeframe(self):
        timeframe = self.chart.topbar['timeframe'].value
        symbol = self.chart.topbar['symbol'].value
        contract = database.Contracts.get(symbol=symbol)
        dataframe = await get_bar_data(symbol, timeframe, 'chart')
        self.chart.set(DataFrame(Candles(database, contract, timeframe, 'chart').get()))
        if not dataframe.empty:
            return
        while True:
            self.chart.update(Series(Candles(database, contract, timeframe, 'last').get()))
            await asyncio.sleep(1)

async def main():
    api = API()

    chart = Chart(api=api, volume_enabled=False, topbar=True, searchbox=True, toolbox=True, debug=True)
    chart.legend(True)
    chart.precision(6)

    symbols = [contract.symbol for contract in database.Contracts.select()]
    timeframes = list(settings.Settings().TIMEFRAMES)

    chart.topbar.textbox('symbol', symbols[0])
    chart.topbar.switcher('timeframe', api.on_timeframe, *timeframes, default=timeframes[0])

    contract = database.Contracts.select().where(database.Contracts.symbol == symbols[0]).get()
    chart.set(DataFrame(Candles(database, contract, timeframes[0], 'chart').get()))

    await chart.show_async(block=False)
    while True:
        chart.update(Series(Candles(database, contract, timeframes[0], 'last').get()))
        await asyncio.sleep(1)

if __name__ == '__main__':
    asyncio.run(main())

[Zartexo]

Hi @jlixfeld,

can you provide the 12h dataframe as csv file to check if anything is wrong with it?

[louisnw01]

can you provide the 12h dataframe as csv file to check if anything is wrong with it?

Thanks @Zartexo, yes @jlixfeld a minimal reproducible example would be great.

Louis

[jlixfeld]

Indeed. Here's a '5m' which renders correctly and the '12h' which is one that does not.

USDJPY_5m.csv USDJPY_12h.csv

[louisnw01]

Indeed. Here's a '5m' which renders correctly and the '12h' which is one that does not.

USDJPY_5m.csv USDJPY_12h.csv

Hi,

I took a look at the USDJPY_12hr data and it doesnt appear to occur at intervals of 12 hours. These are the first few rows of the time column:

2008-08-29 21:00:00+00:00
2008-09-30 21:00:00+00:00
2008-10-31 21:00:00+00:00
2008-11-28 21:00:00+00:00
2008-12-31 21:00:00+00:00
2009-01-30 21:00:00+00:00
2009-02-27 21:00:00+00:00
2009-03-31 21:00:00+00:00
2009-04-30 21:00:00+00:00

[jlixfeld]

Yes, at the moment the database returns aggregated data the further back in time you go, which is what you are seeing. This is indeed an issue I will address, but as it is, the USDJPY_5m data also has the same rows and it renders ok. In fact, all of the timeframes that do render, all have aggregated data from higher timeframes further back in time.

Also, I wanted to point out that the higher timeframes that do render show the timescale as midnight, which may not be the way it's expected to display. Case in point, IBKR's API returns data relative to 21:00UTC for timeframes longer than 1 hour. That is, 4 hour timeframes are relative to 21:00, 01:00, 05:00, 09:00, etc. 6 hour timeframes are relative to 21:00, 03:00, 09:00, etc. Is there a way to for the time scale to honour the timestamps in the database?

[louisnw01]

The library calculates the timescale based on the most common interval within the dataset. Taking a look at your 5min data, 5 minutes is the most common interval which is why it renders with no problems.

In your '12h' data, only 39 out of 1,437 rows are of a 12 hour interval, so the library doesn't recognise this as 12hr.

Timeframes higher than or equal to 1D will show up as midnight, but anything less will use the timestamps within the dataset.

Louis

[jlixfeld]

As suggested, applying more filters during construction of the dataframe to remove data from other timeframes resolved the issue. Thanks for your time @louisnw01