Closed jhonabreul closed 11 months ago
On Streaming Data > Key Concepts:
Each request has a one-second overhead, so you should package your custom data to minimize requests. Bundle dates together where possible to speed up execution.
because making requests over the internet is an option now. We can recommend using the Object Store for speed.
On Custom Securities > CSV Format Example, Custom Securities > JSON Format Example, Custom Universes > CSV Format Example, and Custom Universes > JSON Format Example, add H3 after "Data Format" (e.g.: "Saving Data") to inform that the data can be saved to the Object Store.
All Define Custom Types should have:
def GetSource(self, config: SubscriptionDataConfig, date: datetime, isLive: bool) -> SubscriptionDataSource:
if not isLive:
return SubscriptionDataSource(CustomDataKey,
SubscriptionTransportMedium.ObjectStore,
FileFormat.Csv)
return SubscriptionDataSource("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1",
SubscriptionTransportMedium.RemoteFile)
to show the ObjectStore usage as "default" for backtest.
On Bulk Downloads, add losing data to use with the ObjectStore in "Recommended Use Cases".
After Download Files, add a section teaching how to use Download+ObjectStore.Save to save a zip file to be used in streaming.
CustomDataKey = "CustomData20231010/ExampleCustomData.zip"
data = self.Download("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1")
self.ObjectStore.SaveBytes(CustomDataKey,
Compression.ZipBytes(bytes(self.CustomData, 'utf-8'), 'data'))
Closed by #1511
Expected Behavior
Document how to use the object store as a data source using:
The linked demo algorithms showcase how to source data stored in the object store and how to use it both in algorithms and research environments, by using the new subscription transport medium
SubscriptionTransportMedium.ObjectStore
. They show case two use cases, streaming the data for the algorithms and in history requests.See new regression algorithms in https://github.com/QuantConnect/Lean/pull/7493
Actual Behavior
New feature, not documented.
Checklist
master
branch