QuantConnect / Lean

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)
https://lean.io
Apache License 2.0
8.96k stars 3.14k forks source link

Alpaca API doesn't exit gracefully at market close #4209

Closed auwsom closed 4 years ago

auwsom commented 4 years ago

Hello, as the title says. I assume other brokerage APIs exit in a way that doesn't loop websocket errors, or will this timeout? It should exit so that the container that it is running in can still be used. I may be able to manually kill the mono job at trading day end as a workaround.

Using Docker Hub image with CSharp BasicAlgo in Debug. (sorry for the horrible formatting)

Ah.. and then I get another error when trying to run it again. I will paste the at the very end, but it may need a separate Issue, because this is the third separate build I've seen it in. I happens the second time I run the Launcher.exe. It looks something to do with the data storage and retrieval [T]. It does not occur with backtesting. But I'm pretty sure if I build yet another image from scratch it will run the first time but not the second. That is with only changes to the config.js for the 'environment' to 'live-alpaca' and setting the two Alpaca API keys.

EDIT: so the Release version runs without the websocket error loop, which makes sense if the Debug is verbosely reporting them. It also isn't breaking on the second run. I will try again after it has been run with live data connection tomorrow and update.

Here are the errors as Gists for readability: error 1 error 2 20200317 19:59:10.561 Trace:: LiveTradingResultHandler.SendStatusUpdate(): Running 20200317 19:59:10.581 Trace:: AlgorithmManager.Run(): Begin DataStream - Start: 3/17/2020 12:00:00 AM Stop: 12/31/2050 12:00:00 AM 20200317 19:59:10.587 Trace:: ProcessVolatilityHistoryRequirements(): Updating volatility models with historical data... 20200317 19:59:10.670 Trace:: Debug: Launching analysis for BasicTemplateFrameworkAlgorithm with LEAN Engine v2.4.0.0 20200317 19:59:11.137 Trace:: AlpacaBrokerage.Subscribe(): SPY 20200317 19:59:11.138 Trace:: DataManager.AddSubscription(): Added SPY,SPY,Minute,TradeBar,Trade,Adjusted. Start: 3/17/2020 7:59:11 PM. End: 12/31/2050 5:00:00 AM 20200317 19:59:11.146 Trace:: DataManager.AddSubscription(): Added SPY,SPY,Minute,TradeBar,Trade,Adjusted. Start: 3/17/2020 7:58:00 PM. End: 12/31/2050 5:00:00 AM 20200317 19:59:11.146 Trace:: UniverseSelection.AddPendingInternalDataFeeds(): Adding internal benchmark data feed SPY,SPY,Minute,TradeBar,Trade,Adjusted 20200317 20:00:00.190 Trace:: BrokerageTransactionHandler.Process(): 03/17/2020 20:00:00 UTC: Submit Order: (1) - OrderId: 1 (BrokerId: ) New MarketOnOpen order for 386 units of SPY Status: Unprocessed 20200317 20:00:00.194 Trace:: Total margin information: TotalMarginUsed: 0.00, MarginRemaining: 98197.42 20200317 20:00:00.199 Trace:: Order request margin information: MarginUsed: 0.00MarginRemaining: 98197.42 20200317 20:00:00.359 Trace:: BrokerageTransactionHandler.OrderStatusChanged(): Time: 03/17/2020 20:00:00 OrderID: 1 Symbol: SPY Status: Invalid Message: Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET BrokerI d: 20200317 20:00:00.199 Trace:: Order request margin information: MarginUsed: 0.00MarginRemaining: 98197.42 20200317 20:00:00.359 Trace:: BrokerageTransactionHandler.OrderStatusChanged(): Time: 03/17/2020 20:00:00 OrderID: 1 Symbol: SPY Status: Invalid Message: Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET BrokerI d: 20200317 20:00:00.365 Trace:: LiveTradingResultHandler.OrderEvent(): Time: 03/17/2020 20:00:00 OrderID: 1 Symbol: SPY Status: Invalid Message: Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET 20200317 20:00:00.366 Trace:: Brokerage.OnMessage(): Warning - Code: -1 - Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET 20200317 20:00:00.460 Trace:: Debug: New Order Event: Time: 03/17/2020 20:00:00 OrderID: 1 Symbol: SPY Status: Invalid Message: Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET 20200317 20:00:01.262 ERROR:: Brokerage Warning: Error placing order: OPG order rejected, must be submitted after 7:00pm and before 09:28am ET 20200317 20:00:11.045 Trace:: Isolator.ExecuteWithTimeLimit(): Used: 35, Sample: 62, App: 319, CurrentTimeStepElapsed: 00:00.000 Stack overflow in unmanaged: IP: 0x6dbf32, fault addr: 0x7f7818106ff8 Stack overflow: IP: 0x7f784137e4fa, fault addr: 0x7f78180feff0 Stacktrace: at <unknown> <0xffffffff> at System.Net.Sockets.Socket.QueueIOSelectorJob (System.Threading.SemaphoreSlim,intptr,System.IOSelectorJob) [0x0003e] in <a360a10e096b4659a9142a244ddaea3e>:0 <...> at System.Net.Sockets.Socket.BeginReceive (byte[],int,int,System.Net.Sockets.SocketFlags,System.Net.Sockets.SocketError&,System.AsyncCallback,object) [0x0005c] in <a360a10e096b4659a9142a244ddaea3e>:0 at System.Net.Sockets.Socket.BeginReceive (byte[],int,int,System.Net.Sockets.SocketFlags,System.AsyncCallback,object) [0x00000] in <a360a10e096b4659a9142a244ddaea3e>:0 at System.Net.Sockets.NetworkStream.BeginRead (byte[],int,int,System.AsyncCallback,object) [0x0009b] in <a360a10e096b4659a9142a244ddaea3e>:0 at System.IO.Stream/<>c.<BeginEndReadAsync>b__43_0 (System.IO.Stream,System.IO.Stream/ReadWriteParameters,System.AsyncCallback,object) [0x00000] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 at System.Threading.Tasks.TaskFactory1.FromAsyncTrim<System.IO.Stream, System.IO.Stream/ReadWriteParameters> (System.IO.Stream,System.IO.Stream/ReadWriteParameters,System.Func5<System.IO.Stream, System.IO.Stream/ReadWriteParameters, System.AsyncC allback, object, System.IAsyncResult>,System.Func3<System.IO.Stream, System.IAsyncResult, int>) [0x00008] in :0 at System.IO.Stream.BeginEndReadAsync (byte[],int,int) [0x00000] in :0 at System.IO.Stream.ReadAsync (byte[],int,int,System.Threading.CancellationToken) [0x00009] in :0 at Mono.Net.Security.MobileAuthenticatedStream/d66.MoveNext () [0x00094] in :0 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start<Mono.Net.Security.MobileAuthenticatedStream/d66> (Mono.Net.Security.MobileAuthenticatedStream/d66&) [0x0002c] in :0 at Mono.Net.Security.MobileAuthenticatedStream.InnerRead (bool,int,System.Threading.CancellationToken) [0x0003b] in :0 at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.InnerRead (bool,int,System.Threading.CancellationToken) [0x00034] in :0 at Mono.Net.Security.AsyncProtocolRequest/d25.MoveNext () [0x00040] in :0 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1<System.Nullable1>.Start<Mono.Net.Security.AsyncProtocolRequest/d25> (Mono.Net.Security.AsyncProtocolRequest/d25&) [0x0002c] in :0 at Mono.Net.Security.AsyncProtocolRequest.InnerRead (System.Threading.CancellationToken) [0x0002b] in :0 at Mono.Net.Security.AsyncProtocolRequest/d24.MoveNext () [0x0002f] in :0 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Mono.Net.Security.AsyncProtocolRequest/d24> (Mono.Net.Security.AsyncProtocolRequest/d24&) [0x0002c] in :0 at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken) [0x0002b] in :0 at Mono.Net.Security.AsyncProtocolRequest/d23.MoveNext () [0x0002a] in :0 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1<Mono.Net.Security.AsyncProtocolResult>.Start<Mono.Net.Security.AsyncProtocolRequest/<StartOperation>d__23> (Mono.Net.Security.AsyncProtocolRequest/<StartOperation>d__23&) [0x0002c] in <b0e1ad75 73a24fd5a9f2af9595e677e7>:0 at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken) [0x0002b] in <a360a10e096b4659a9142a244ddaea3e>:0 at Mono.Net.Security.MobileAuthenticatedStream/<StartOperation>d__58.MoveNext () [0x000b4] in <a360a10e096b4659a9142a244ddaea3e>:0 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.Start<Mono.Net.Security.MobileAuthenticatedStream/d58> (Mono.Net.Security.MobileAuthenticatedStream/d58&) [0x0002c] in <b0e1ad7573a24fd5a9f2af9595e677e7

:0 at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream/OperationType,Mono.Net.Security.AsyncProtocolRequest,System.Threading.CancellationToken) [0x0003b] in :0 at Mono.Net.Security.MobileAuthenticatedStream.BeginRead (byte[],int,int,System.AsyncCallback,object) [0x00013] in :0 at System.Net.Security.SslStream.BeginRead (byte[],int,int,System.AsyncCallback,object) [0x00006] in :0 at WebSocketSharp.Ext.ReadBytesAsync (System.IO.Stream,int,System.Action1<byte[]>,System.Action1) [0x0005c] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readHeaderAsync (System.IO.Stream,System.Action1<WebSocketSharp.WebSocketFrame>,System.Action1) [0x0000e] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.ReadFrameAsync (System.IO.Stream,bool,System.Action1<WebSocketSharp.WebSocketFrame>,System.Action1) [0x00023] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocket/<>cDisplayClass170_0.b0 () [0x00000] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocket/<>cDisplayClass170_0.b1 (WebSocketSharp.WebSocketFrame) [0x00044] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame/<>cDisplayClass82_0.b3 (WebSocketSharp.WebSocketFrame) [0x00012] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame/<>cDisplayClass77_0.b0 (byte[]) [0x00035] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.Ext/<>cDisplayClass54_0.b0 (System.IAsyncResult) [0x0009e] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task,System.AsyncCallback,object) [0x00014] in :0 at Mono.Net.Security.MobileAuthenticatedStream.BeginRead (byte[],int,int,System.AsyncCallback,object) [0x00018] in :0 at System.Net.Security.SslStream.BeginRead (byte[],int,int,System.AsyncCallback,object) [0x00006] in :0 at WebSocketSharp.Ext.ReadBytesAsync (System.IO.Stream,int,System.Action1<byte[]>,System.Action1) [0x0005c] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readPayloadDataAsync (System.IO.Stream,WebSocketSharp.WebSocketFrame,System.Action1<WebSocketSharp.WebSocketFrame>,System.Action1) [0x00097] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame/<>cDisplayClass82_0.b2 (WebSocketSharp.WebSocketFrame) [0x00000] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readMaskingKeyAsync (System.IO.Stream,WebSocketSharp.WebSocketFrame,System.Action1<WebSocketSharp.WebSocketFrame>,System.Action1) [0x0004a] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame/<>c__DisplayClass82_0.b__1 (WebSocketSharp.WebSocketFrame) [0x00000] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readExtendedPayloadLengthAsync (System.IO.Stream,WebSocketSharp.WebSocketFrame,System.Action1<WebSocketSharp.WebSocketFrame>

20200317 20:14:15.287 Trace:: Config.GetValue(): debug-mode - Using default value: False 20200317 20:14:15.297 Trace:: Config.Get(): Configuration key not found. Key: plugin-directory - Using default value: 20200317 20:14:15.300 Trace:: Config.Get(): Configuration key not found. Key: composer-dll-directory - Using default value: /root/Lean/Launcher/bin/Debug/ 20200317 20:14:15.357 Trace:: Python for .NET Assembly: Python.Runtime, Version=1.0.5.29, Culture=neutral, PublicKeyToken=null 20200317 20:14:15.373 Trace:: Python for .NET Assembly: nPython, Version=1.0.5.29, Culture=neutral, PublicKeyToken=null 20200317 20:14:15.377 Trace:: Config.Get(): Configuration key not found. Key: log-handler - Using default value: CompositeLogHandler 20200317 20:14:15.404 Trace:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/ 20200317 20:14:15.406 Trace:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 20200317 20:14:15.406 Trace:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: ../../../Data/ 20200317 20:14:15.406 Trace:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.4.0.0 Mode: DEBUG (64bit) 20200317 20:14:15.407 Trace:: Engine.Main(): Started 8:14 PM 20200317 20:14:15.409 Trace:: Config.Get(): Configuration key not found. Key: api-access-token - Using default value: 20200317 20:14:15.412 Trace:: Config.GetValue(): job-user-id - Using default value: 0 20200317 20:14:15.412 Trace:: Config.GetValue(): job-project-id - Using default value: 0 20200317 20:14:15.413 Trace:: Config.Get(): Configuration key not found. Key: algorithm-language - Using default value: Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Must specify valid information for parsing in the string. at System.Enum+EnumResult.SetFailure (System.Enum+ParseFailureKind failure, System.String failureMessageID, System.Object failureMessageFormatArgument) [0x00023] in :0 at System.Enum.TryParseEnum (System.Type enumType, System.String value, System.Boolean ignoreCase, System.Enum+EnumResult& parseResult) [0x00077] in :0 at System.Enum.Parse (System.Type enumType, System.String value, System.Boolean ignoreCase) [0x00010] in :0 at System.Enum.Parse (System.Type enumType, System.String value) [0x00000] in :0 at QuantConnect.Queues.JobQueue..ctor () [0x00040] in :0 at (wrapper managed-to-native) System.Reflection.MonoCMethod.InternalInvoke(System.Reflection.MonoCMethod,object,object[],System.Exception&) at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00002] in :0 --- End of inner exception stack trace --- at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00014] in :0 at System.RuntimeType.CreateInstanceMono (System.Boolean nonPublic) [0x000a8] in :0 at System.RuntimeType.CreateInstanceSlow (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00009] in :0 at System.RuntimeType.CreateInstanceDefaultCtor (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00027] in :0 at System.Activator.CreateInstance (System.Type type, System.Boolean nonPublic) [0x00020] in :0 at System.Activator.CreateInstance (System.Type type) [0x00000] in :0 at QuantConnect.Util.Composer.GetExportedValueByTypeName[T] (System.String typeName) [0x000c3] in :0 at QuantConnect.Lean.Engine.LeanEngineSystemHandlers.FromConfiguration (QuantConnect.Util.Composer composer) [0x00011] in <6def4409f4ed4a548d84a0f6c71a0dc0>:0 at QuantConnect.Lean.Launcher.Program.Main (System.String[] args) [0x00113] in <80168248cd9d4a8693fd5fb8958c9ff8>:0 [ERROR] FATAL UNHANDLED EXCEPTION: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Must specify valid information for parsing in the string. at System.Enum+EnumResult.SetFailure (System.Enum+ParseFailureKind failure, System.String failureMessageID, System.Object failureMessageFormatArgument) [0x00023] in :0 at System.Enum.TryParseEnum (System.Type enumType, System.String value, System.Boolean ignoreCase, System.Enum+EnumResult& parseResult) [0x00077] in :0 at System.Enum.Parse (System.Type enumType, System.String value, System.Boolean ignoreCase) [0x00010] in :0 at System.Enum.Parse (System.Type enumType, System.String value) [0x00000] in :0 at QuantConnect.Queues.JobQueue..ctor () [0x00040] in :0 at (wrapper managed-to-native) System.Reflection.MonoCMethod.InternalInvoke(System.Reflection.MonoCMethod,object,object[],System.Exception&) at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00002] in :0 --- End of inner exception stack trace --- at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00014] in :0 at System.RuntimeType.CreateInstanceMono (System.Boolean nonPublic) [0x000a8] in :0 at System.RuntimeType.CreateInstanceSlow (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00009] in :0 at System.RuntimeType.CreateInstanceDefaultCtor (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00027] in :0 at System.Activator.CreateInstance (System.Type type, System.Boolean nonPublic) [0x00020] in :0 at System.Activator.CreateInstance (System.Type type) [0x00000] in :0 at QuantConnect.Util.Composer.GetExportedValueByTypeName[T] (System.String typeName) [0x000c3] in :0 at QuantConnect.Lean.Engine.LeanEngineSystemHandlers.FromConfiguration (QuantConnect.Util.Composer composer) [0x00011] in <6def4409f4ed4a548d84a0f6c71a0dc0>:0 at QuantConnect.Lean.Launcher.Program.Main (System.String[] args) [0x00113] in <80168248cd9d4a8693fd5fb8958c9ff8>:0

auwsom commented 4 years ago

I solved my secondary errors. The original title may still refer to the websocket Stack overflow in unmanaged: IP: 0x6dbf32, fault addr: 0x7f7818106ff8error when in Debug, which may be happening in Release.

However, the Error 1 from above was caused by not changing the algo file back from Python when setting the language back to CSharp in config.json for debugging. And Error 2 seems to be caused when I created a bash alias for the Launcher.exe which cant be called from the root dir: "mono /root/Lean/Launcher/bin/Release/QuantConnect.Lean.Launcher.exe".

I should have used a subshell for the alias: ( cd /root/Lean/Launcher/bin/Debug/ ; mono ./QuantConnect.Lean.Launcher.exe )

gsalaz98 commented 4 years ago

Hey @auwsom!

I've tried replicating your runtime environment as much as possible, but am not able to replicate the issue in my development environment.

Could you please provide the following details? It'll help me resolve this issue more quickly.

Thanks!

jaredbroad commented 4 years ago

Closing pending reproduce test case