search

DaedalicEntertainment / ue4-test-automation

Facilitates setting up integration test suits with Unreal Engine 4 Gauntlet.
https://www.daedalic.com
MIT License
215 stars 62 forks source link

Gauntlet: Array dimensions exceeded supported range #28

Closed janousch closed 2 years ago

janousch commented 2 years ago

After testing >300 tests we get the following error from Gauntlet:

Error: Array dimensions exceeded supported range..

   at System.String.SplitInternal(Char[] separator, Int32 count, StringSplitOptions options)
   at Gauntlet.UnrealLogParser.CreateSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 205
   at Gauntlet.UnrealLogParser.GetSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 189
   at Gauntlet.UnrealRoleArtifacts.get_LogSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealSession.cs:line 424
   at Gauntlet.UnrealTestNode`1.<>c.<GetWarnings>b__8_0(UnrealRoleArtifacts A) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs:line 116
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.<>c.<ReportMasterSummary>b__22_1(TestExecutionInfo T) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at System.Linq.Enumerable.WhereListIterator`1.MoveNext()
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.ReportMasterSummary(Int32 CurrentPass, Int32 NumPasses, TimeSpan Duration, IEnumerable`1 AllInfo) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at Gauntlet.TextExecutor.ExecuteTests(TestExecutorOptions InOptions, IEnumerable`1 RequiredTests) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 440
   at Gauntlet.RunUnreal.ExecuteTests(UnrealTestOptions Options, IEnumerable`1 TestList) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:line 307
AutomationTool exiting with ExitCode=152 (Error_TestFailure)
janousch commented 2 years ago

The problem is that we have an AnimationModifier class defined in C++ inside of an editor module. When we run the automated tests with Gauntlet we don't have access to that module. When the test runs, Unreal uses/opens the editor animation sequences, inside of the AnimSequence we got the blueprint version of that AnimationModifier and then we get the log message that the parent is missing. The logs are filling up the context string of the test artifact. When the UnrealLogParser tries to split that string he fails.

janousch commented 2 years ago

Fixed in https://github.com/janousch/ue4-test-automation/tree/develop I added the option to execute console commands before starting the tests. By suppressing the logs about missing references to editor only objects the size of the log file decreased by several GB and the Error disappeared. Currently, there are doubts whether these changes should be merged.