QuantConnect / Documentation

QuantConnect Wiki Style Documentation Behind QuantConnect
https://www.quantconnect.com/docs/v2/
Apache License 2.0
171 stars 135 forks source link

Add section about object store data source #1500

Closed jhonabreul closed 11 months ago

jhonabreul commented 1 year ago

Expected Behavior

Document how to use the object store as a data source using:

The linked demo algorithms showcase how to source data stored in the object store and how to use it both in algorithms and research environments, by using the new subscription transport medium SubscriptionTransportMedium.ObjectStore. They show case two use cases, streaming the data for the algorithms and in history requests.

See new regression algorithms in https://github.com/QuantConnect/Lean/pull/7493

Actual Behavior

New feature, not documented.

Checklist

AlexCatarino commented 11 months ago

On Streaming Data > Key Concepts:

because making requests over the internet is an option now. We can recommend using the Object Store for speed.

On Custom Securities > CSV Format Example, Custom Securities > JSON Format Example, Custom Universes > CSV Format Example, and Custom Universes > JSON Format Example, add H3 after "Data Format" (e.g.: "Saving Data") to inform that the data can be saved to the Object Store.

All Define Custom Types should have:

    def GetSource(self, config: SubscriptionDataConfig, date: datetime, isLive: bool) -> SubscriptionDataSource:
        if not isLive:
            return SubscriptionDataSource(CustomDataKey, 
                SubscriptionTransportMedium.ObjectStore, 
                FileFormat.Csv)
        return SubscriptionDataSource("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1",
            SubscriptionTransportMedium.RemoteFile)

to show the ObjectStore usage as "default" for backtest.

On Bulk Downloads, add losing data to use with the ObjectStore in "Recommended Use Cases".

After Download Files, add a section teaching how to use Download+ObjectStore.Save to save a zip file to be used in streaming.

CustomDataKey = "CustomData20231010/ExampleCustomData.zip"
data = self.Download("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1")
self.ObjectStore.SaveBytes(CustomDataKey,
    Compression.ZipBytes(bytes(self.CustomData, 'utf-8'), 'data'))
AlexCatarino commented 11 months ago

Closed by #1511