QuantConnect / Lean

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)
https://lean.io
Apache License 2.0
8.92k stars 3.12k forks source link

BaseCollection Datasets Aren't Structured Well in DataFrames #6630

Closed DerekMelchin closed 1 day ago

DerekMelchin commented 1 year ago

Expected Behavior

If we make a history request for a dataset that uses BaseCollection, the DataFrame has columns for each data property.

Actual Behavior

If we make a history request for a dataset that uses BaseCollection, the DataFrame just has a data column, which is a list. image

Potential Solution

If the custom type is BaseCollection, iterate it to build the DataFrame rows.

Reproducing the Problem

qb = QuantBook()
symbol = qb.AddEquity("AAPL").Symbol
dataset_symbol = qb.AddData(QuiverCongress, symbol).Symbol
qb.History(dataset_symbol, datetime(2022, 1, 1), datetime(2022, 9, 15))

System Information

QC Cloud

Checklist

Martin-Molinero commented 3 weeks ago

Hm this should be covered already by https://github.com/QuantConnect/Lean/blob/master/Algorithm/QCAlgorithm.Python.cs#L1575 TryCleanupCollectionDataFrame might need checking out why isn't it not working for this case, should first add unit test reproducing this issue