search

Lombiq / UI-Testing-Toolbox

Web UI testing toolbox for Orchard Core applications with Selenium. It discovers so many issues that you'll get impostor syndrome.
BSD 3-Clause "New" or "Revised" License
17 stars 6 forks source link

Randomly failing UI tests, when upgrading to OC 1.5 (OSOE-483) #239

Closed dministro closed 1 year ago

dministro commented 1 year ago

In some cases, the browser navigation results in HTTP 500. In this state, there are no logs under FaliureDump folder, and looks like the web app won't start.

Related:

Jira issue

dministro commented 1 year ago

After I collected the logs of many failed runs, I found errors only in the internal-nlog.txt, like below:

...
2022-11-24 20:18:24.4539 Error Failed to add type 'NLog.Web.LayoutRenderers.AspNetRequestConnectionIdLayoutRenderer'. Exception: System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at NLog.Config.Factory`2.RegisterDefinition(String itemName, Type itemDefinition, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.RegisterType(Type type, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.ScanTypes(Type[] types, String assemblyName, String itemNamePrefix)
2022-11-24 20:18:24.4540 Error Failed to add type 'NLog.Web.LayoutRenderers.AspNetRequestConnectionIdLayoutRenderer'. Exception: System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at NLog.Config.Factory`2.RegisterDefinition(String itemName, Type itemDefinition, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.RegisterType(Type type, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.ScanTypes(Type[] types, String assemblyName, String itemNamePrefix)
2022-11-24 20:18:24.4625 Error Failed to add type 'NLog.Web.LayoutRenderers.AspNetMvcActionRenderer'. Exception: System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at NLog.Config.Factory`2.RegisterDefinition(String itemName, Type itemDefinition, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.RegisterType(Type type, String assemblyName, String itemNamePrefix)
   at NLog.Config.Factory`2.ScanTypes(Type[] types, String assemblyName, String itemNamePrefix)
...

I started from here and added a lock around Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory.CreateHost(IHostBuilder builder) in Lombiq.Tests.UI.Services.OrchardCoreHosting.OrchardApplicationFactory.CreateHost(IHostBuilder builder), because the CreateHost invokes Microsoft.Extensions.Hosting.HostFactoryResolver.CreateHost() which starts the web application in a new thread, and finally this can result in exceptions as I found in the internal-nlog.txt in case of parallel execution of tests.

After I added the change above, I wasn't faced with the issue again. I ran the tests locally and on GHA without any issues. See: https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/pull/307

0liver commented 1 year ago

Great work, Sherlock! 🧐

Piedone commented 1 year ago

Fixed by https://github.com/Lombiq/UI-Testing-Toolbox/pull/240.