Sample Test Fails When Dynamics Asks For Feedback

[davegoopot]

Describe the bug

Running the basicgallery sample with dotnet run results in a timeout. Running with dotnet run -l trace works just fine

To Reproduce

Steps to reproduce the behavior:

1. dotnet run -l trace
2. That works
3. dotnet run
4. Throws the exception stack below

``c:\code\PowerApps-TestEngine\src\PowerAppsTestEngine\bin\Debug\net6.0.playwright\package\lib\server\dispatchers\dispatcher.js:281 stack: validMetadata.stack, ^

TypeError: Cannot read properties of undefined (reading 'stack') at DispatcherConnection.dispatch (c:\code\PowerApps-TestEngine\src\PowerAppsTestEngine\bin\Debug\net6.0.playwright\package\lib\server\dispatchers\dispatcher.js:281:28) at PipeTransport.transport.onmessage (c:\code\PowerApps-TestEngine\src\PowerAppsTestEngine\bin\Debug\net6.0.playwright\package\lib\cli\driver.js:61:57) at Immediate. (c:\code\PowerApps-TestEngine\src\PowerAppsTestEngine\bin\Debug\net6.0.playwright\package\lib\protocol\transport.js:89:34) at processImmediate (node:internal/timers:464:21) [Error]: Waiting timed out.

[Error]: System.TimeoutException: The operation has timed out. at Microsoft.PowerApps.TestEngine.Helpers.PollingHelper.CheckIfTimedOut(DateTime startTime, Int32 timeout, ILogger logger) in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\Helpers\PollingHelper.cs:line 74 at Microsoft.PowerApps.TestEngine.Helpers.PollingHelper.PollAsync[T](T value, Func2 conditionToCheck, Func1 functionToCall, Int32 timeout, ILogger logger) in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\Helpers\PollingHelper.cs:line 44 at Microsoft.PowerApps.TestEngine.PowerApps.PowerAppFunctions.LoadPowerAppsObjectModelAsync() in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\PowerApps\PowerAppFunctions.cs:line 142 at Microsoft.PowerApps.TestEngine.PowerFx.PowerFxEngine.UpdatePowerFxModelAsync() in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\PowerFx\PowerFxEngine.cs:line 142 at Microsoft.PowerApps.TestEngine.SingleTestRunner.RunTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in C:\Users\david.potts\OneDrive - British Engineering Services Limited\code_bes\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\SingleTestRunner.cs:line 112

[Error]: Connection closed (Process exited)

Unhandled exception. Microsoft.Playwright.PlaywrightException: Connection closed (Process exited) at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Object args) in //src/Playwright/Transport/Connection.cs:line 107 at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal) in //src/Playwright/Transport/Connection.cs:line 479 at Microsoft.Playwright.Core.BrowserContext.CloseAsync() in /_/src/Playwright/Core/BrowserContext.cs:line 175 at Microsoft.PowerApps.TestEngine.TestInfra.PlaywrightTestInfraFunctions.EndTestRunAsync() in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\TestInfra\PlaywrightTestInfraFunctions.cs:line 226
at Microsoft.PowerApps.TestEngine.SingleTestRunner.RunTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in C:\Users\david.potts\OneDrive - British Engineering Services Limited\code_bes\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\SingleTestRunner.cs:line 197 at Microsoft.PowerApps.TestEngine.TestEngine.RunOneTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 143 at Microsoft.PowerApps.TestEngine.TestEngine.RunTestByWorkerCountAsync(String testRunId, String testRunDirectory, String domain, String queryParams) in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 136 at Microsoft.PowerApps.TestEngine.TestEngine.RunTestAsync(String testConfigFile, String environmentId, String tenantId, String outputDirectory, String domain, String queryParams) in c:\code\PowerApps-TestEngine\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 99 at Program.

$(String[] args) in c:\code\PowerApps-TestEngine\src\PowerAppsTestEngine\Program.cs:line 160 at Program.

(String[] args)``

[davegoopot]

I've discovered that if I add this slowMo setting into src\Microsoft.PowerApps.TestEngine\TestInfra\PlaywrightTestInfraFunctions.cs then the tests run fine. Is there a way I can pass this setting in the test definition file?

[davegoopot]

Actually I was wrong on the above. Even the SlowMo doesn't seem to work reliably for me.

[landonmsft]

Thank you for reaching out, Dave! I'm looking into this now. Could you post your config.json file?

[davegoopot]

Nothing much to it:

{
  "environmentId": "ENVIRONMENT-GUID",
  "tenantId": "TENANT-GUID",
  "testPlanFile": "../../samples/basicgallery/testPlan.fx.yaml",
  "outputDirectory": "daveresults"
}

[landonmsft]

If you change it to this, and try dotnet run, does that work?:

{
  "environmentId": "ENVIRONMENT-GUID",
  "tenantId": "TENANT-GUID",
  "testPlanFile": "../../samples/basicgallery/testPlan.fx.yaml",
  "outputDirectory": "daveresults",
  "logLevel": ""
}

[landonmsft]

The reason I ask is, this issue is not showing up on my machine when attempting to replicate your issue. There must be something deeper going on, but figured to try the quick possible solutions first to rule them out. If what I posted above and changing what I posted above to have "logLevel": "Trace" also doesn't work, could you post your TestSettings and your testPlan.fx.Yaml file?

You can also email them to landonrogers (at) microsoft (dot) com

Thank you!

[kagarlickij]

@landonarogers I have the same issue When I add "logLevel": "" I get different error:

   at Microsoft.PowerApps.TestEngine.PowerFx.PowerFxEngine.UpdatePowerFxModelAsync() in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\PowerFx\PowerFxEngine.cs:line 142
   at Microsoft.PowerApps.TestEngine.SingleTestRunner.RunTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\SingleTestRunner.cs:line 112

[Error]: Connection closed (Process exited)

Unhandled exception. Microsoft.Playwright.PlaywrightException: Connection closed (Process exited)
   at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Object args) in /_/src/Playwright/Transport/Connection.cs:line 107
   at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal) in /_/src/Playwright/Transport/Connection.cs:line 479
   at Microsoft.Playwright.Core.BrowserContext.CloseAsync() in /_/src/Playwright/Core/BrowserContext.cs:line 175
   at Microsoft.PowerApps.TestEngine.TestInfra.PlaywrightTestInfraFunctions.EndTestRunAsync() in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\TestInfra\PlaywrightTestInfraFunctions.cs:line 226
   at Microsoft.PowerApps.TestEngine.SingleTestRunner.RunTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\SingleTestRunner.cs:line 197
   at Microsoft.PowerApps.TestEngine.TestEngine.RunOneTestAsync(String testRunId, String testRunDirectory, TestSuiteDefinition testSuiteDefinition, BrowserConfiguration browserConfig, String domain, String queryParams) in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 143
   at Microsoft.PowerApps.TestEngine.TestEngine.RunTestByWorkerCountAsync(String testRunId, String testRunDirectory, String domain, String queryParams) in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 136
   at Microsoft.PowerApps.TestEngine.TestEngine.RunTestAsync(String testConfigFile, String environmentId, String tenantId, String outputDirectory, String domain, String queryParams) in D:\a\1\s\tests\src\Microsoft.PowerApps.TestEngine\TestEngine.cs:line 99
   at Program.<Main>$(String[] args) in D:\a\1\s\tests\src\PowerAppsTestEngine\Program.cs:line 160
   at Program.<Main>(String[] args)
##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 3762504530
##[warning].NET 5 has some compatibility issues with older Nuget versions(<=5.7), so if you are using an older Nuget version(and not dotnet cli) to restore, then the dotnet cli commands (e.g. dotnet build) which rely on such restored packages might fail. To mitigate such error, you can either: (1) - Use dotnet cli to restore, (2) - Use Nuget version 5.8 to restore, (3) - Use global.json using an older sdk version(<=3) to build
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
##[error]Dotnet command failed with non-zero exit code on the following projects : 
Finishing: run tests

[landonmsft]

That's interesting; looking into that now. In the meantime, could you try running on .NET SDK 6.0.402? I wonder if the warning is related to the issue.

[davegoopot]

The reason I ask is, this issue is not showing up on my machine when attempting to replicate your issue. There must be something deeper going on, but figured to try the quick possible solutions first to rule them out. If what I posted above and changing what I posted above to have "logLevel": "Trace" also doesn't work, could you post your TestSettings and your testPlan.fx.Yaml file?

You can also email them to landonrogers (at) microsoft (dot) com

Thank you!

No luck with adding in the "logLevel" as above. Works neither with blank nor "Trace" logLevels.

testPlan.fx.Yaml attached. TestSettings are inline in the testPlan file. testPlan.fx.yaml.zip

[davegoopot]

@landonarogers I think I've found the problem. See the updated title and description above. Here's the key image showing the problem:

[landonmsft]

Thank you! That's very helpful; I'll see if I can replicate this, and look into resolving this problem in TestEngine. Thanks again!

[davegoopot]

The app is the basic gallery one from the samples folder of the git repo.

[landonmsft]

Hey, Dave! We've identified a potential fix to account for this popup. I don't have a specific timeframe to give you on when exactly this will be integrated into the codebase but wanted to keep you in the loop on this!

[davegoopot]

Thanks @landonarogers . Any hints for a work around in the meantime. At the moment I'm just putting up with sporadic failures, but would be great if there was something I could do to reduce those!

[landonmsft]

I'm not sure of any temporary workarounds; the popup is designed to only query you periodically, so it will unfortunately continue to occur.

However, you could manually reduce your timeout wait-time so that it doesn't take so long on every failure, so that it'll fail fast. You'll want to find a time that won't take long, but also a time that's not too short to where your testPlan won't wrongly timeout when it should succeed.

[landonmsft]

Here's the documentation on how to input your desired wait-time. If you aren't currently doing this, the default is 30 seconds. 5-10 might be more manageable! :)

[landonmsft]

Hey, Dave! Could you try to reproduce this now, and let me know if the issue still persists?

[davegoopot]

I'll find some time to give it a try. Presumably I need to pull the latest code from git?

On Thu, 16 Feb 2023 at 06:12, Landon Rogers @.***> wrote:

Hey, Dave! Could you try to reproduce this now, and let me know if the issue still persists?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/PowerApps-TestEngine/issues/169#issuecomment-1432580193, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5SDWXN7JDM6AJQPD2UWB3WXXALPANCNFSM6AAAAAARVP2W34 . You are receiving this because you authored the thread.Message ID: @.***>

[landonmsft]

Yes, please! Thank you! :)