Bug using ToolKit to query complex ticker names: Unknown brokerage symbol

antoinedray commented 3 years ago

Expected Behavior

Running the following command should run without errors (ex: runs well for --tickers=tBTCUSD):

dotnet QuantConnect.ToolBox.dll --app=BFXDL --from-date="20171101-00:00:00" --tickers=t1INCH:USD --resolution=Minute

Should work well as the ticker is listed in Lean/Data/symbol-properties/symbol-properties-database.csv

Actual Behavior

dotnet QuantConnect.ToolBox.dll --app=BFXDL --from-date="20171101-00:00:00" --tickers=t1INCH:USD --resolution=Minute

Gives the following error:

20211005 13:47:50.546 TRACE:: Config.GetValue(): debug-mode - Using default value: False
20211005 13:47:50.553 TRACE:: Config.Get(): Configuration key not found. Key: results-destination-folder - Using default value: 
20211005 13:47:50.553 TRACE:: Config.Get(): Configuration key not found. Key: plugin-directory - Using default value: 
20211005 13:47:50.556 TRACE:: Config.Get(): Configuration key not found. Key: composer-dll-directory - Using default value: 
20211005 13:47:50.561 TRACE:: Composer(): Loading Assemblies from Lean/ToolBox/bin/Debug/
20211005 13:47:50.598 TRACE:: Config.Get(): Configuration key not found. Key: log-handler - Using default value: CompositeLogHandler
20211005 13:47:50.636 TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
20211005 13:47:50.637 TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 
20211005 13:47:50.637 TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: ../../../Data/
20211005 13:47:50.656 TRACE:: Config.Get(): Configuration key not found. Key: map-file-provider - Using default value: LocalDiskMapFileProvider
20211005 13:47:50.878 TRACE:: BaseWebSocketsBrokerage.Connect(): Connecting...
20211005 13:47:50.879 TRACE:: WebSocketClientWrapper connection task started: wss://api.bitfinex.com/ws/2
20211005 13:47:50.880 TRACE:: WebSocketClientWrapper.HandleConnection(wss://api.bitfinex.com/ws/2): Connecting...
20211005 13:47:56.301 TRACE:: WebSocketClientWrapper.OnOpen(): Connection opened (IsOpen:True, State:Open): wss://api.bitfinex.com/ws/2
20211005 13:47:56.301 TRACE:: BaseWebsocketsBrokerage(): WebSocket.Open. Subscribing
20211005 13:47:56.347 TRACE:: WebSocketClientWrapper connection task ended: wss://api.bitfinex.com/ws/2
20211005 13:47:56.347 TRACE:: WebSocketClientWrapper.OnClose(): Connection closed (IsOpen:False, State:CloseSent): wss://api.bitfinex.com/ws/2
20211005 13:47:56.360 ERROR:: BitfinexDownloaderProgram.BitfinexDownloader():  System.ArgumentException: Unknown brokerage symbol: 1INCH
   at QuantConnect.Brokerages.SymbolPropertiesDatabaseSymbolMapper.GetLeanSymbol(String brokerageSymbol, SecurityType securityType, String market, DateTime expirationDate, Decimal strike, OptionRight optionRight) in Lean/Brokerages/SymbolPropertiesDatabaseSymbolMapper.cs:line 120
   at QuantConnect.ToolBox.BitfinexDownloader.BitfinexDataDownloader.GetSymbol(String ticker) in Lean/ToolBox/BitfinexDownloader/BitfinexDataDownloader.cs:line 92
   at QuantConnect.ToolBox.BitfinexDownloader.BitfinexDownloaderProgram.BitfinexDownloader(IList`1 tickers, String resolution, DateTime fromDate, DateTime toDate) in Lean/ToolBox/BitfinexDownloader/BitfinexDownloaderProgram.cs:line 54

Potential Solution

I've narrowed the issue to the ApplicationParser class (Lean/Configuration/ApplicationParser.cs) that treats the ":" as a character to get nested objects (line 65). Thus coming out of the parser, the ticker is not read as "t1INCH:USD" but as two tickers "t1INCH" and "USD".

Reproducing the Problem

Running the following command should be enough to reproduce the error:

dotnet QuantConnect.ToolBox.dll --app=BFXDL --from-date="20171101-00:00:00" --tickers=t1INCH:USD --resolution=Minute

System Information

Hardware Overview:

Model Name: iMac Pro Model Identifier: iMacPro1,1 Processor Name: 8-Core Intel Xeon W Processor Speed: 3.2 GHz Number of Processors: 1 Total Number of Cores: 8 L2 Cache (per Core): 1 MB L3 Cache: 11 MB Hyper-Threading Technology: Enabled Memory: 32 GB System Firmware Version: 1554.140.20.0.0 (iBridge: 18.16.14759.0.1,0) Activation Lock Status: Disabled

Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues
[x] I have provided detailed steps to reproduce the issue

Martin-Molinero commented 3 years ago

Hey @antoinedray! Thanks for the report,

--tickers=t1INCH:USD

Does using --tickers=t1INCHUSD work instead?

antoinedray commented 3 years ago

Hello @Martin-Molinero !

Thank you for your fast response !

Just tried, but I get the same error:

$ dotnet QuantConnect.ToolBox.dll --app=BFXDL --from-date="20171101-00:00:00" --tickers=t1INCHUSD --resolution=Minute

20211008 15:30:01.867 TRACE:: Config.GetValue(): debug-mode - Using default value: False
20211008 15:30:01.883 TRACE:: Config.Get(): Configuration key not found. Key: results-destination-folder - Using default value: 
20211008 15:30:01.884 TRACE:: Config.Get(): Configuration key not found. Key: plugin-directory - Using default value: 
20211008 15:30:01.887 TRACE:: Config.Get(): Configuration key not found. Key: composer-dll-directory - Using default value: 
20211008 15:30:01.894 TRACE:: Composer(): Loading Assemblies from Lean/ToolBox/bin/Debug/
20211008 15:30:01.944 TRACE:: Config.Get(): Configuration key not found. Key: log-handler - Using default value: CompositeLogHandler
20211008 15:30:01.989 TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
20211008 15:30:01.990 TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 
20211008 15:30:01.990 TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: ../../../Data/
20211008 15:30:02.014 TRACE:: Config.Get(): Configuration key not found. Key: map-file-provider - Using default value: LocalDiskMapFileProvider
20211008 15:30:02.339 TRACE:: BaseWebSocketsBrokerage.Connect(): Connecting...
20211008 15:30:02.340 TRACE:: WebSocketClientWrapper connection task started: wss://api.bitfinex.com/ws/2
20211008 15:30:02.341 TRACE:: WebSocketClientWrapper.HandleConnection(wss://api.bitfinex.com/ws/2): Connecting...
20211008 15:30:07.967 TRACE:: WebSocketClientWrapper.OnOpen(): Connection opened (IsOpen:True, State:Open): wss://api.bitfinex.com/ws/2
20211008 15:30:07.967 TRACE:: BaseWebsocketsBrokerage(): WebSocket.Open. Subscribing
20211008 15:30:08.023 TRACE:: WebSocketClientWrapper connection task ended: wss://api.bitfinex.com/ws/2
20211008 15:30:08.024 TRACE:: WebSocketClientWrapper.OnClose(): Connection closed (IsOpen:False, State:CloseSent): wss://api.bitfinex.com/ws/2
20211008 15:30:08.049 ERROR:: BitfinexDownloaderProgram.BitfinexDownloader():  System.ArgumentException: Unknown brokerage symbol: t1INCHUSD
   at QuantConnect.Brokerages.SymbolPropertiesDatabaseSymbolMapper.GetLeanSymbol(String brokerageSymbol, SecurityType securityType, String market, DateTime expirationDate, Decimal strike, OptionRight optionRight) in Lean/Brokerages/SymbolPropertiesDatabaseSymbolMapper.cs:line 120
   at QuantConnect.ToolBox.BitfinexDownloader.BitfinexDataDownloader.GetSymbol(String ticker) in Lean/ToolBox/BitfinexDownloader/BitfinexDataDownloader.cs:line 92
   at QuantConnect.ToolBox.BitfinexDownloader.BitfinexDownloaderProgram.BitfinexDownloader(IList`1 tickers, String resolution, DateTime fromDate, DateTime toDate) in Lean/ToolBox/BitfinexDownloader/BitfinexDownloaderProgram.cs:line 54

Martin-Molinero commented 2 years ago

Hey @antoinedray! Sorry for the delay with this one, moving forward --tickers will accept Lean tickers, for brokerage symbol t1INCH:USD should directly pass in 1INCHUSD

antoinedray commented 2 years ago

Thank you so much @Martin-Molinero Amazing work 🔥

QuantConnect / Lean