search

DaedalicEntertainment / ue4-test-automation

Facilitates setting up integration test suits with Unreal Engine 4 Gauntlet.
https://www.daedalic.com
MIT License
218 stars 64 forks source link

Throw Error When Using Gauntlet to Run Test #12

Closed WuAlan closed 3 years ago

WuAlan commented 3 years ago

Followed the readme document to run a demo test case by Gauntlet, but get an error, what this mean and what should I do? image

Run Commond: .\RunUAT.bat RunUnreal -project="G:\DemoProject\Demo1\Demo1.uproject" -scriptdir="G:\DemoProject\Demo1" -platform=Win64 -configuration=Development -build=editor -test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest"

Automation.ParseCommandLine: Parsing command line: RunUnreal -project=G:\DemoProject\Demo1\Demo1.uproject -scriptdir=G:\DemoProject\Demo1 -platform=Win64 -configuration=Development -build=editor -test=DaedalicTestAutomationPlugin.Automation.DaeGauntletTest Automation.Process: Setting up command environment. InternalUtils.SafeFileExists: SafeFileExists D:\UE4\UE_4.23\Engine\Binaries\DotNET\AutomationTool.exe=True InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT= InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT= InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_EngineSavedFolder= CommandUtils.SetEnvVar: SetEnvVar uebp_EngineSavedFolder=D:/UE4/UE_4.23/Engine/Programs/AutomationTool/Saved InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_EngineSavedFolder=D:/UE4/UE_4.23/Engine/Programs/AutomationTool/Saved InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_CSVFile= InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LogFolder= InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT=D:/UE4/UE_4.23 CommandUtils.SetEnvVar: SetEnvVar uebp_LogFolder=C:/Users/yanlin.wu/AppData/Roaming/Unreal Engine/AutomationTool/Logs/D+UE4+UE_4.23 InternalUtils.SafeDeleteFile: SafeDeleteFile C:\Users\yanlin.wu\AppData\Roaming\Unreal Engine\AutomationTool\Logs\D+UE4+UE_4.23\Log.txt InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_FinalLogFolder= CommandUtils.SetEnvVar: SetEnvVar uebp_FinalLogFolder=C:/Users/yanlin.wu/AppData/Roaming/Unreal Engine/AutomationTool/Logs/D+UE4+UE_4.23 InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\robocopy.exe=True InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\mount.exe=False InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\Sysnative\mount.exe=False InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\cmd.exe=True CommandUtils.SetEnvVar: SetEnvVar MallocNanoZone=0 InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_UATChildInstance=0 WindowsPlatform.FindVSInstallDirs: Found Visual Studio installation: f:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise (Product=Microsoft.VisualStudio.Product.Enterprise, Version=15.9.28307.718, Sort=0) AndroidPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.AndroidPlatformSDK has no valid SDK IOSPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.IOSPlatformSDK using manually installed SDK HTML5PlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.HTML5PlatformSDK using manually installed SDK WindowsPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.WindowsPlatformSDK using manually installed SDK Log.OutputMessage: Added device Win64:default to pool Log.WriteException: ============================================================================== Log.WriteException: ERROR: Unable to find type DaedalicTestAutomationPlugin.Automation.DaeGauntletTest in assemblies. Namespaces= System.Linq.Enumerable+WhereSelectArrayIterator2[System.String,System.String]. Log.WriteException: (see C:\Users\yanlin.wu\AppData\Roaming\Unreal Engine\AutomationTool\Logs\D+UE4+UE_4.23\Log.txt for full exception trace) Log.WriteException: Log.WriteException: AutomationException: Unable to find type DaedalicTestAutomationPlugin.Automation.DaeGauntletTest in assemblies. Namespaces= System.Linq.Enumerable+WhereSelectArrayIterator2[System.String,System.String]. Log.WriteException: At Gauntlet.Utils.TestConstructor.GetTypeForTest(String TestName, IEnumerable1 Namespaces) Locate: D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.Utils.cs:Line: 540 Log.WriteException: At Gauntlet.Utils.TestConstructor.ConstructTest[TestType,ParamType](String TestName, ParamType Arg, IEnumerable1 Namespaces) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.Utils.cs:Line: 575 Log.WriteException: At Gauntlet.RunUnreal.CreateTestList(UnrealTestContext Context, Params DefaultParams, ArgumentWithParams PlatformParams) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 366 Log.WriteException: At Gauntlet.RunUnreal.RunTests(UnrealTestOptions ContextOptions) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 237 Log.WriteException: At Gauntlet.RunUnreal.Execute() Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 56 Log.WriteException: At AutomationTool.Automation.Execute(List1 CommandsToExecute, Dictionary2 Commands) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:Line: 551 Log.WriteException: At AutomationTool.Automation.Process(String[] Arguments, StartupTraceListener StartupListener) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:Line: 521 Log.WriteException: At AutomationTool.Program.MainProc(String[] Arguments, StartupTraceListener StartupListener) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 176 Log.WriteException: At AutomationTool.Program.<>c__DisplayClass1_0.

b__2() Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 88 Log.WriteException: At AutomationTool.InternalUtils.RunSingleInstance(Func`1 Main) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Utils.cs:Line: 725 Log.WriteException: At AutomationTool.Program.Main(String[] Arguments) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 88 Log.WriteException: ============================================================================== Program.Main: AutomationTool exiting with ExitCode=1 (Error_Unknown)

npruehs commented 3 years ago

Hey @WuAlan !

Can you confirm you've performed step 4 of https://github.com/DaedalicEntertainment/ue4-test-automation#adding-the-plugin , more specifically

"Copy the DaedalicTestAutomationPlugin.Automation folder to the Build/Scripts folder next to your .uproject file."?

Can you see the respective C# project next to your AutomationTool in Visual Studio?

I assume you're using the source version of Unreal Engine?

WuAlan commented 3 years ago

Thanks for reply, Found I did not use source code version of Unreal, I'll try it again, Thanks!

WuAlan commented 3 years ago

Hey @WuAlan !

Can you confirm you've performed step 4 of https://github.com/DaedalicEntertainment/ue4-test-automation#adding-the-plugin , more specifically

"Copy the DaedalicTestAutomationPlugin.Automation folder to the Build/Scripts folder next to your .uproject file."?

Can you see the respective C# project next to your AutomationTool in Visual Studio?

I assume you're using the source version of Unreal Engine?

Hey, @npruehs , I succuss runned the test from RunUAT.bat now, but it seems no junit report generated followed the tuturial, where should be wrong here?

commandline: D:\UE4\UnrealEngine-4.26.0-release\Engine\Build\BatchFiles\RunUAT.bat RunUnreal -project="G:\ue4test\autotestdemo1\autotestdemo1.uproject" -scriptdir="G:\ue4test\autotestdemo1" -platform=Win64 -configuration=Development -build=editor -test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest(JUnitReportPath=D:\test\junit-report.xml,ReportPath=D:\test\reports)"

Console output: D:\UE4\UnrealEngine-4.26.0-release\Engine\Build\BatchFiles>D:\UE4\UnrealEngine-4.26.0-release\Engine\Build\BatchFiles\RunUAT.bat RunUnreal -project="G:\ue4test\autotestdemo1\autotestdemo1.uproject" -scriptdir="G:\ue4test\autotestdemo1" -platform=Win64 -configuration=Development -build=editor -test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest(JUnitReportPath=D:\test\junit-report.xml,ReportPath=D:\test\reports)" Running AutomationTool... Parsing command line: RunUnreal -project=G:\ue4test\autotestdemo1\autotestdemo1.uproject -scriptdir=G:\ue4test\autotestdemo1 -platform=Win64 -configuration=Development -build=editor -test=DaedalicTestAutomationPlugin.Automation.DaeGauntletTest(JUnitReportPath=D:\test\junit-report.xml,ReportPath=D:\test\reports) -compile Dependencies are up to date (0.291s). Skipping compile. Added device Win64:default to pool Preparing to start 1 automation test(s) Starting test pass 1 of 1 Acquiring device LocalDevice0 Selected devices LocalDevice0 for client(s). Prepping Test DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) is ready to run Status: Completed:0, Running:0, Starting: 1, Waiting:0 Starting Test DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) Launching Editor on LocalDevice0 Launched test DaedalicTestAutomationPlugin.Automation.DaeGauntletTest at 3:42:01 LogGauntlet: Display: Gauntlet Initialized LogGauntlet: Display: Added Gauntlet controller DaeGauntletTestController LogGauntlet: Display: Changed state to Gauntlet_Initialized LogGauntlet: Display: GauntletHeartbeat: Idle LogGauntlet: Display: Changed state to Gauntlet_DiscoveringTests LogGauntlet: Display: Changed state to Gauntlet_Running LogGauntlet: Display: Changed state to Gauntlet_Finished LogGauntlet: Display: TEST COMPLETE. EXIT CODE: 0 LogGauntlet: Display: Gauntlet Shutdown *

Finished Test: DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) in 00:07 Saving artifacts to D:\UE4\UnrealEngine-4.26.0-release\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest_(Win64_DevelopmentEditorGame) Wrote Log to D:\UE4\UnrealEngine-4.26.0-release\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest(Win64_Development_EditorGame)\EditorGame\EditorGameOutput.log Skipping archival of assets for editor EditorGame DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) result=Passed

* Test DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) Passed Test Time: 00:13 (Waited:00:00, Setup:00:00) Completed test pass 1 of 1.

1 of 1 Tests Passed in 00:14. (0 Failed, 0 Passed with Warnings)

DaedalicTestAutomationPlugin.Automation.DaeGauntletTest (Win64 Development EditorGame) result=Passed

DaedalicTestAutomationPlugin.Automation.DaeGauntletTest Report

Process Role: EditorGame (Win64 Development)

Result: Reason=Process exited with code 0, Code=0

FatalErrors: 0, Ensures: 0, Errors: 0, Warnings: 2, Hash: 0

Artifacts

Log: D:\UE4\UnrealEngine-4.26.0-release\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest_(Win64_Development_EditorGame)\EditorGame\EditorGameOutput.log

Commandline: G:\ue4test\autotestdemo1\autotestdemo1.uproject -gauntlet=DaeGauntletTestController -ResX=1280 -ResY=720 -windowed -unattended -stdout -AllowStdOutLogVerbosity -gauntlet.heartbeatperiod=30 -multihome=10.0.75.1 -nomcp -notimeouts -noepicportal -report.xml -game -userdir="D:\UE4\UnrealEngine-4.26.0-release\GauntletTemp\DeviceCache\Win64\LocalDevice0_UserDir"

D:\UE4\UnrealEngine-4.26.0-release\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest_(Win64_Development_EditorGame)\EditorGame

Summary

DaedalicTestAutomationPlugin.Automation.DaeGauntletTest Passed

Context: Win64 Development EditorGame

FatalErrors: 0, Ensures: 0, Errors: 0, Warnings: 2

Result: Passed, ResultHash: 0

BUILD SUCCESSFUL AutomationTool exiting with ExitCode=0 (Success)

WuAlan commented 3 years ago

I finally solve this problem by add a slash before parameters like: test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest(-JUnitReportPath=D:\test\junit-report.xml,-ReportPath=D:\test\reports)"