search

tgstation / tgstation-server

A production scale tool for DreamMaker server management
https://tgstation.github.io/tgstation-server/
GNU Affero General Public License v3.0
66 stars 82 forks source link

Spurious Integration Test Errors #1206

Closed Cyberboss closed 3 years ago

Cyberboss commented 3 years ago

We get a lot of failed tests that shouldn't because of what I can only imagine to be a timing issue in their writing. It's not reproducible locally (With my specs at least: 3.78GHz, 16GB RAM).

The problem is in the Tgstation.Server.Tests.Instance.WatchdogTest test class. That class does a lot of chat/topic/interop/process fuckery to thoroughly exercise the watchdog.

See https://github.com/tgstation/tgstation-server/blob/dev/.github/CONTRIBUTING.md#development-environment for how to setup the integration tests locally.

I'm putting a standing 50$ CAD bounty for whoever is able to fix this issue without losing test functionality.

Cyberboss commented 3 years ago

Logs:

2021-01-28T02:36:10.7343638Z  TEST: WATCHDOG BYOND VERSION UPDATE TEST
2021-01-28T02:36:10.7345074Z  [02:33:45] dbg: Tgstation.Server.Host.Controllers.ByondController (Instance:1|Job:|Request:GET /Byond|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7346864Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7348955Z  [02:33:45] dbg: Tgstation.Server.Host.Controllers.DreamDaemonController (Instance:1|Job:|Request:GET /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7351080Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7353578Z  [02:33:45] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:|Request:GET /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7f3b2dda-7f27-405d-b365-62c38bd386d0)
2021-01-28T02:36:10.7355512Z      Compile job 7 lock count now: 1
2021-01-28T02:36:10.7357363Z  [02:33:45] wrn: Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider (Instance:1|Job:|Request:|User:|Monitor:|Bridge:10|Chat:|IR:)
2021-01-28T02:36:10.7359139Z      Error sending discord message!
2021-01-28T02:36:10.7359957Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7361357Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7363033Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7364654Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7366778Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7369525Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7372442Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7375234Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7378722Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7383834Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendMessage(UInt64 channelId, String message, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 364
2021-01-28T02:36:10.7387587Z  [02:33:45] wrn: Tgstation.Server.Host.Components.Chat.ChatManager (Instance:1|Job:30|Request:PUT /DreamMaker|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7389012Z      Error sending deploy message to provider 2!
2021-01-28T02:36:10.7389831Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7391111Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7392675Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7394163Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7396125Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7398969Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7401670Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7404296Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7407636Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7414018Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendUpdateMessage(RevisionInformation revisionInformation, Version byondVersion, Nullable`1 estimatedCompletionTime, String gitHubOwner, String gitHubRepo, UInt64 channelId, Boolean localCommitPushed, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 449
2021-01-28T02:36:10.7419674Z     at Tgstation.Server.Host.Components.Chat.ChatManager.<>c__DisplayClass33_0.<<QueueDeploymentMessage>b__0>d.MoveNext() in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/ChatManager.cs:line 712
2021-01-28T02:36:10.7422760Z  [02:33:48] dbg: Tgstation.Server.Host.Controllers.DreamDaemonController (Instance:1|Job:|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7424810Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7426749Z  [02:33:48] dbg: Tgstation.Server.Host.Jobs.JobManager (Instance:1|Job:|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7428133Z      Registering job 34: Launch DreamDaemon...
2021-01-28T02:36:10.7429961Z  [02:33:48] vrb: Tgstation.Server.Host.Jobs.JobManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7431177Z      Starting job...
2021-01-28T02:36:10.7433247Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7434941Z      Begin LaunchImplNoLock
2021-01-28T02:36:10.7437005Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Chat.ChatManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7438591Z      Chat send "WD: Launching..." to channels: 1, 2
2021-01-28T02:36:10.7440694Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7442354Z      Compile job 7 lock count now: 2
2021-01-28T02:36:10.7444426Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7446185Z      Initializing controller with CompileJob 7...
2021-01-28T02:36:10.7449225Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Deployment.Remote.RemoteDeploymentManagerFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7451892Z      Creating remote deployment manager for remote git provider GitHub...
2021-01-28T02:36:10.7454946Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Deployment.Remote.GitHubRemoteDeploymentManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7457551Z      Not updating deployment as it is missing a repo ID or deployment ID.
2021-01-28T02:36:10.7459816Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7461681Z      Prep for server launch. pendingSwappable is not available
2021-01-28T02:36:10.7464113Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7465967Z      Loading compile job 7...
2021-01-28T02:36:10.7468103Z  [02:33:48] vrb: Tgstation.Server.Host.Components.StaticFiles.Configuration (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7469993Z      No event scripts starting with "WatchdogLaunch" detected
2021-01-28T02:36:10.7472177Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7473809Z      Compile job 7 lock count now: 3
2021-01-28T02:36:10.7475988Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7477459Z      Hard linking compile job...
2021-01-28T02:36:10.7479521Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Session.SessionControllerFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7481172Z      Begin session launch...
2021-01-28T02:36:10.7483021Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Chat.ChatTrackingContext (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7484447Z      Activated
2021-01-28T02:36:10.7486185Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Byond.ByondManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7487690Z      Creating ByondExecutableLock lock for version 513.1536.1
2021-01-28T02:36:10.7489913Z  [02:33:48] dbg: Tgstation.Server.Host.Components.Session.SessionControllerFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7491643Z      Launching session with CompileJob 7...
2021-01-28T02:36:10.7493769Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Session.SessionControllerFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7495386Z      Bind test: 35837
2021-01-28T02:36:10.7496904Z  [02:33:48] dbg: Tgstation.Server.Host.System.ProcessExecutor (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7500628Z      Launching process in /tmp/TGS4_INTEGRATION_TEST/ffaba028-e399-43b5-b308-23b75d18be4c/64563a26-e324-422e-bfe4-8d2e9ff08c38/Game/Live: /tmp/TGS4_INTEGRATION_TEST/ffaba028-e399-43b5-b308-23b75d18be4c/64563a26-e324-422e-bfe4-8d2e9ff08c38/Byond/513.1536.1/byond/bin/DreamDaemon.sh tests/DMAPI/LongRunning/long_running_test_copy.dmb -port 35837 -ports 1-65535 -close -safe -public -params "server_service_version=5.3.0&tgs_port=5010&tgs_key=rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS%2fwEz32O0"
2021-01-28T02:36:10.7503775Z  [02:33:48] vrb: Tgstation.Server.Host.System.Process (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7504876Z      Created process ID: 14275
2021-01-28T02:36:10.7506533Z  [02:33:48] vrb: Tgstation.Server.Host.Components.InstanceManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7508198Z      Registered bridge handler: rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS/wEz32O0
2021-01-28T02:36:10.7510815Z  [02:33:48] vrb: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7512975Z      Waiting for LaunchResult based on initial bridge request with a timeout of 60s...
2021-01-28T02:36:10.7515659Z  [02:33:48] dbg: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7518101Z      Created session controller. CommsKey: rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS/wEz32O0, Port: 35837
2021-01-28T02:36:10.7520368Z  [02:33:48] wrn: Tgstation.Server.Host.System.Process (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7521814Z      Unable to raise process priority for PID 14275!
2021-01-28T02:36:10.7522896Z  System.ComponentModel.Win32Exception (13): Permission denied
2021-01-28T02:36:10.7524417Z     at System.Diagnostics.Process.set_PriorityClassCore(ProcessPriorityClass value)
2021-01-28T02:36:10.7526022Z     at System.Diagnostics.Process.set_PriorityClass(ProcessPriorityClass value)
2021-01-28T02:36:10.7528704Z     at Tgstation.Server.Host.System.Process.SetHighPriority() in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/System/Process.cs:line 173
2021-01-28T02:36:10.7531715Z  [02:33:48] dbg: Tgstation.Server.Host.System.Process (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7533211Z      Error on WaitForInputIdle()!
2021-01-28T02:36:10.7534489Z  System.InvalidOperationException: WaitForInputIdle failed.  This could be because the process does not have a graphical interface.
2021-01-28T02:36:10.7536281Z     at System.Diagnostics.Process.WaitForInputIdleCore(Int32 milliseconds)
2021-01-28T02:36:10.7537886Z     at System.Diagnostics.Process.WaitForInputIdle()
2021-01-28T02:36:10.7540473Z     at Tgstation.Server.Host.System.Process.<>c__DisplayClass16_0.<.ctor>b__0() in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/System/Process.cs:line 93
2021-01-28T02:36:10.7542913Z  [02:33:49] dbg: Tgstation.Server.Host.Controllers.JobController (Instance:1|Job:|Request:GET /Job/34|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7544804Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7547434Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7549329Z      Launch result: Exit Code: , Time ms
2021-01-28T02:36:10.7551591Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7553436Z      Controller initialization failure!
2021-01-28T02:36:10.7554744Z  Tgstation.Server.Host.Jobs.JobException: DreamDaemon failed to start within the configured timeout!
2021-01-28T02:36:10.7556729Z   ---> Tgstation.Server.Host.Jobs.JobException: Server timed out on startup: 60s
2021-01-28T02:36:10.7558020Z     --- End of inner exception stack trace ---
2021-01-28T02:36:10.7561732Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.CheckLaunchResult(ISessionController controller, String serverName, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 472
2021-01-28T02:36:10.7567780Z     at Tgstation.Server.Host.Components.Watchdog.BasicWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/BasicWatchdog.cs:line 235
2021-01-28T02:36:10.7572467Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7574403Z      DisposeAndNullControllers
2021-01-28T02:36:10.7576884Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7578769Z      Disposing...
2021-01-28T02:36:10.7580540Z  [02:35:42] vrb: Tgstation.Server.Host.System.Process (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7581990Z      Terminating PID 14275...
2021-01-28T02:36:10.7583372Z  [02:35:42] vrb: Tgstation.Server.Host.Controllers.BridgeController (Instance:|Job:|Request:|User:|Monitor:|Bridge:22|Chat:|IR:)
2021-01-28T02:36:10.7585525Z      Bridge Request: {"minimumSecurityLevel":2,"version":"5.3.0","customCommands":[],"commandType":1,"accessIdentifier":"rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS/wEz32O0"}
2021-01-28T02:36:10.7587869Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:|Request:|User:|Monitor:|Bridge:22|Chat:|IR:)
2021-01-28T02:36:10.7589352Z      Handling bridge request...
2021-01-28T02:36:10.7590927Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Session.SessionController (Instance:1|Job:|Request:|User:|Monitor:|Bridge:22|Chat:|IR:)
2021-01-28T02:36:10.7592587Z      ApiValidationStatus set to RequiresUltrasafe
2021-01-28T02:36:10.7594329Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Chat.ChatTrackingContext (Instance:1|Job:|Request:|User:|Monitor:|Bridge:22|Chat:|IR:)
2021-01-28T02:36:10.7595842Z      Custom commands set.
2021-01-28T02:36:10.7597199Z  [02:35:42] vrb: Tgstation.Server.Host.Controllers.BridgeController (Instance:|Job:|Request:|User:|Monitor:|Bridge:22|Chat:|IR:)
2021-01-28T02:36:10.7605221Z      Bridge Response: {"runtimeInformation":{"serverVersion":"4.8.2","serverPort":5010,"instanceName":"IntegrationTestInstance","revision":{"commitSha":"87db5e20b85bf9de47e8482ef793434aecdcaf18","timestamp":"2021-01-28T02:31:24+00:00","originCommitSha":"380b6e089eb24a7a39297a0cda1b5f9469a3d230"},"securityLevel":1,"testMerges":[{"timeMerged":"0","pullRequestRevision":"ed7c11a0f1a53485dc9f5366d393c46eda8a75f6","targetCommitSha":"ed7c11a0f1a53485dc9f5366d393c46eda8a75f6","revision":{"commitSha":"87db5e20b85bf9de47e8482ef793434aecdcaf18","timestamp":"2021-01-28T02:31:24+00:00","originCommitSha":"380b6e089eb24a7a39297a0cda1b5f9469a3d230"},"titleAtMerge":"Never merge this PR","bodyAtMerge":"","url":"https://github.com/tgstation/tgstation-server/pull/958","author":"Cyberboss","number":958,"comment":"asdffdsa"}],"channels":[{"id":"1","friendlyName":"***","connectionName":"irc.rizon.net","tag":"butt2"},{"id":"2","friendlyName":"bot-testing","connectionName":"Dextraspace","isAdminChannel":1,"tag":"butt"}]}}
2021-01-28T02:36:10.7611135Z  [02:35:42] vrb: Tgstation.Server.Host.System.Process (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7612412Z      PID 14275 exited with code 137
2021-01-28T02:36:10.7614755Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Session.SessionControllerFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7616726Z      DreamDaemon Output:
2021-01-28T02:36:10.7617212Z  Thu Jan 28 02:33:48 2021
2021-01-28T02:36:10.7617715Z  World opened on network port 35837.
2021-01-28T02:36:10.7618317Z  Welcome BYOND! (5.0 Public Version 513.1536)
2021-01-28T02:36:10.7618900Z  Initial value of sleep_offline: 1
2021-01-28T02:36:10.7619497Z  Info: Activating API for version 5.3.0
2021-01-28T02:36:10.7621195Z  Export: http://127.0.0.1:5010/Bridge?data=%7b%22minimumSecurityLevel%22%3a2%2c%22version%22%3a%225.3.0%22%2c%22customCommands%22%3a%5b%5d%2c%22commandType%22%3a1%2c%22accessIdentifier%22%3a%22rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS%2fwEz32O0%22%7d
2021-01-28T02:36:10.7622798Z  The BYOND hub reports that port 35837 is not reachable.
2021-01-28T02:36:10.7624735Z  [02:35:42] vrb: Tgstation.Server.Host.System.ProcessExecutor (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7626296Z      PID 14275 termination event completed
2021-01-28T02:36:10.7628290Z  [02:35:42] vrb: Tgstation.Server.Host.Components.InstanceManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7630195Z      Unregistered bridge handler: rQZBNLEJOVPnQdeM3l90sQGnhNVcEYhS/wEz32O0
2021-01-28T02:36:10.7632642Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7634275Z      Compile job 7 lock count now: 2
2021-01-28T02:36:10.7636426Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Deployment.DmbFactory (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7638088Z      Compile job 7 lock count now: 1
2021-01-28T02:36:10.7640174Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7641842Z      Unhardlinking compile job...
2021-01-28T02:36:10.7643680Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Watchdog.PosixWatchdog (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7645146Z      Failed to start watchdog!
2021-01-28T02:36:10.7646143Z  Tgstation.Server.Host.Jobs.JobException: DreamDaemon failed to start within the configured timeout!
2021-01-28T02:36:10.7647761Z   ---> Tgstation.Server.Host.Jobs.JobException: Server timed out on startup: 60s
2021-01-28T02:36:10.7648797Z     --- End of inner exception stack trace ---
2021-01-28T02:36:10.7651807Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.CheckLaunchResult(ISessionController controller, String serverName, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 472
2021-01-28T02:36:10.7656691Z     at Tgstation.Server.Host.Components.Watchdog.BasicWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/BasicWatchdog.cs:line 235
2021-01-28T02:36:10.7661523Z     at Tgstation.Server.Host.Components.Watchdog.BasicWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/BasicWatchdog.cs:line 253
2021-01-28T02:36:10.7666652Z     at Tgstation.Server.Host.Components.Watchdog.PosixWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/PosixWatchdog.cs:line 109
2021-01-28T02:36:10.7672692Z     at Tgstation.Server.Host.Components.Watchdog.PosixWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/PosixWatchdog.cs:line 139
2021-01-28T02:36:10.7678988Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.LaunchNoLock(Boolean startMonitor, Boolean announce, Boolean announceFailure, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 393
2021-01-28T02:36:10.7682941Z  [02:35:42] dbg: Tgstation.Server.Host.Controllers.JobController (Instance:1|Job:|Request:GET /Job/34|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7684815Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7687372Z  [02:35:42] vrb: Tgstation.Server.Host.Components.Chat.ChatManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7688979Z      Chat send "WD: Startup failed!" to channels: 1, 2
2021-01-28T02:36:10.7690692Z  [02:35:42] dbg: Tgstation.Server.Host.Jobs.JobManager (Instance:1|Job:34|Request:PUT /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:7c41f87a-f32f-4911-ba1b-26a49ef4c254)
2021-01-28T02:36:10.7691976Z      Job 34 exited with error!
2021-01-28T02:36:10.7693117Z  Tgstation.Server.Host.Jobs.JobException: DreamDaemon failed to start within the configured timeout!
2021-01-28T02:36:10.7694873Z   ---> Tgstation.Server.Host.Jobs.JobException: Server timed out on startup: 60s
2021-01-28T02:36:10.7696059Z     --- End of inner exception stack trace ---
2021-01-28T02:36:10.7699445Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.CheckLaunchResult(ISessionController controller, String serverName, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 472
2021-01-28T02:36:10.7704755Z     at Tgstation.Server.Host.Components.Watchdog.BasicWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/BasicWatchdog.cs:line 235
2021-01-28T02:36:10.7710019Z     at Tgstation.Server.Host.Components.Watchdog.BasicWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/BasicWatchdog.cs:line 253
2021-01-28T02:36:10.7715666Z     at Tgstation.Server.Host.Components.Watchdog.PosixWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/PosixWatchdog.cs:line 109
2021-01-28T02:36:10.7721219Z     at Tgstation.Server.Host.Components.Watchdog.PosixWatchdog.InitControllers(Task chatTask, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/PosixWatchdog.cs:line 139
2021-01-28T02:36:10.7727045Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.LaunchNoLock(Boolean startMonitor, Boolean announce, Boolean announceFailure, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 393
2021-01-28T02:36:10.7732944Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.LaunchNoLock(Boolean startMonitor, Boolean announce, Boolean announceFailure, ReattachInformation reattachInfo, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 425
2021-01-28T02:36:10.7738384Z     at Tgstation.Server.Host.Components.Watchdog.WatchdogBase.Launch(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Watchdog/WatchdogBase.cs:line 900
2021-01-28T02:36:10.7742411Z     at Tgstation.Server.Host.Jobs.JobManager.RunJob(Job job, JobEntrypoint operation, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Jobs/JobManager.cs:line 115
2021-01-28T02:36:10.7745537Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider (Instance:1|Job:33|Request:PUT /DreamMaker|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7747247Z      Error sending discord message!
2021-01-28T02:36:10.7748017Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7749392Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7750959Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7752467Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7754340Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7757012Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7759713Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7762595Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7766230Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7771281Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendMessage(UInt64 channelId, String message, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 364
2021-01-28T02:36:10.7775960Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider (Instance:1|Job:|Request:DELETE /DreamDaemon|User:1|Monitor:|Bridge:|Chat:|IR:36f15de1-9c50-41a0-a8f8-e09b4f7409f1)
2021-01-28T02:36:10.7777934Z      Error sending discord message!
2021-01-28T02:36:10.7778694Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7779994Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7781563Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7783074Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7784946Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7787500Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7790201Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7792795Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7796553Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7802041Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendMessage(UInt64 channelId, String message, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 364
2021-01-28T02:36:10.7806397Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider (Instance:1|Job:|Request:|User:|Monitor:|Bridge:18|Chat:|IR:)
2021-01-28T02:36:10.7808164Z      Error sending discord message!
2021-01-28T02:36:10.7808931Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7810231Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7811808Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7813354Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7814990Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7817291Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7819633Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7822057Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7825683Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7831146Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendMessage(UInt64 channelId, String message, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 364
2021-01-28T02:36:10.7835174Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Chat.ChatManager (Instance:1|Job:33|Request:PUT /DreamMaker|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7836789Z      Error sending deploy message to provider 2!
2021-01-28T02:36:10.7837665Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7839073Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7840757Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7842506Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7844391Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7846945Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7849645Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7852593Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7856022Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7862432Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendUpdateMessage(RevisionInformation revisionInformation, Version byondVersion, Nullable`1 estimatedCompletionTime, String gitHubOwner, String gitHubRepo, UInt64 channelId, Boolean localCommitPushed, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 449
2021-01-28T02:36:10.7868295Z     at Tgstation.Server.Host.Components.Chat.ChatManager.<>c__DisplayClass33_0.<<QueueDeploymentMessage>b__0>d.MoveNext() in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/ChatManager.cs:line 712
2021-01-28T02:36:10.7871828Z  [02:35:42] wrn: Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider (Instance:1|Job:|Request:|User:|Monitor:|Bridge:17|Chat:|IR:)
2021-01-28T02:36:10.7873600Z      Error sending discord message!
2021-01-28T02:36:10.7874427Z  System.TimeoutException: The operation has timed out.
2021-01-28T02:36:10.7875890Z     at Discord.Net.Queue.RequestBucket.EnterAsync(Int32 id, RestRequest request)
2021-01-28T02:36:10.7877596Z     at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
2021-01-28T02:36:10.7879205Z     at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
2021-01-28T02:36:10.7881265Z     at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
2021-01-28T02:36:10.7883905Z     at Discord.API.DiscordRestApiClient.SendJsonAsync[TResponse](String method, String endpoint, Object payload, String bucketId, ClientBucketType clientBucket, RequestOptions options)
2021-01-28T02:36:10.7886590Z     at Discord.API.DiscordRestApiClient.CreateMessageAsync(UInt64 channelId, CreateMessageParams args, RequestOptions options)
2021-01-28T02:36:10.7889189Z     at Discord.Rest.ChannelHelper.SendMessageAsync(IMessageChannel channel, BaseDiscordClient client, String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7892377Z     at Discord.WebSocket.SocketTextChannel.Discord.IMessageChannel.SendMessageAsync(String text, Boolean isTTS, Embed embed, RequestOptions options)
2021-01-28T02:36:10.7897427Z     at Tgstation.Server.Host.Components.Chat.Providers.DiscordProvider.SendMessage(UInt64 channelId, String message, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/src/Tgstation.Server.Host/Components/Chat/Providers/DiscordProvider.cs:line 364
2021-01-28T02:36:10.7901099Z  [02:35:43] dbg: Tgstation.Server.Host.Controllers.JobController (Instance:1|Job:|Request:GET /Job/34|User:1|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7902839Z      Starting API request: Version: 8.3.0. User-Agent: Tgstation.Server.Tests/1.0.0.0
2021-01-28T02:36:10.7905292Z  [01/28/2021 02:35:43 +00:00] TEST ERROR: Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed. DreamDaemon failed to start within the configured timeout!
2021-01-28T02:36:10.7908120Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)
2021-01-28T02:36:10.7910449Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail(String message, Object[] parameters)
2021-01-28T02:36:10.7912428Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail(String message)
2021-01-28T02:36:10.7916605Z     at Tgstation.Server.Tests.Instance.JobsRequiredTest.WaitForJob(Job originalJob, Int32 timeout, Boolean expectFailure, Nullable`1 expectedCode, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/JobsRequiredTest.cs:line 39
2021-01-28T02:36:10.7922724Z     at Tgstation.Server.Tests.Instance.WatchdogTest.RunLongRunningTestThenUpdateWithByondVersionSwitch(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/WatchdogTest.cs:line 392
2021-01-28T02:36:10.7927966Z     at Tgstation.Server.Tests.Instance.WatchdogTest.Run(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/WatchdogTest.cs:line 64
2021-01-28T02:36:10.7931735Z     at Tgstation.Server.Tests.Instance.InstanceTest.RunTests(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/InstanceTest.cs:line 37
2021-01-28T02:36:10.7935115Z     at Tgstation.Server.Tests.IntegrationTest.<>c__DisplayClass7_0.<<TestServer>g__FailFast|0>d.MoveNext() in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/IntegrationTest.cs:line 751
2021-01-28T02:36:10.7936941Z  [02:35:43] inf: Tgstation.Server.Host.Server (Instance:|Job:|Request:|User:|Monitor:|Bridge:|Chat:|IR:)
2021-01-28T02:36:10.7937763Z      Server termination requested!
2021-01-28T02:36:10.7939580Z  [01/28/2021 02:35:43 +00:00] TEST ERROR: Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed. DreamDaemon failed to start within the configured timeout!
2021-01-28T02:36:10.7942332Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)
2021-01-28T02:36:10.7944639Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail(String message, Object[] parameters)
2021-01-28T02:36:10.7946670Z     at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail(String message)
2021-01-28T02:36:10.7950127Z     at Tgstation.Server.Tests.Instance.JobsRequiredTest.WaitForJob(Job originalJob, Int32 timeout, Boolean expectFailure, Nullable`1 expectedCode, CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/JobsRequiredTest.cs:line 39
2021-01-28T02:36:10.7955463Z     at Tgstation.Server.Tests.Instance.WatchdogTest.RunLongRunningTestThenUpdateWithByondVersionSwitch(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/WatchdogTest.cs:line 392
2021-01-28T02:36:10.7960797Z     at Tgstation.Server.Tests.Instance.WatchdogTest.Run(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/WatchdogTest.cs:line 64
2021-01-28T02:36:10.7964600Z     at Tgstation.Server.Tests.Instance.InstanceTest.RunTests(CancellationToken cancellationToken) in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/Instance/InstanceTest.cs:line 37
2021-01-28T02:36:10.7968184Z     at Tgstation.Server.Tests.IntegrationTest.<>c__DisplayClass7_0.<<TestServer>g__FailFast|0>d.MoveNext() in /home/runner/work/tgstation-server/tgstation-server/tests/Tgstation.Server.Tests/IntegrationTest.cs:line 751
2021-01-28T02:36:10.7970260Z  --- End of stack trace from previous location where exception was thrown ---
Cyberboss commented 3 years ago

From the above, a needed HTTP request from BYOND comes in AFTER DreamDaemon is terminated 1 minute later.

Cyberboss commented 3 years ago

Quite honestly, the solution might just be to use self-hosted runners for the linux integration tests