How to run all Microsoft Test Codeunits

[andreasblueher]

Hello Tobias,

starting with your example on how to do automated al testing, I tried to get the Microsoft tests up and running. CALTestManagement.EnableTestToRun in combination with CALCommandLineTestRunner ÓnRun but many tests are failing in Cronus (Container image: microsoft/bcsandbox:us)

Do you have some hints/links/blogs where I can start looking. Am I wrong to expect Microsoft Tests to succeed in Cronus?

Thank you very much

[tfenster]

Hello Andreas,

everything that needs a GUI or client callback will fail, so I would guess a lot of standard tests will have that problem. What are the failure messages you get?

[andreasblueher]

I'll post a couple of them, although I'm not sure if that's really helpful. So basically what you're saying is that those tests are not supposed to be run on Azure DevOps or automatically in general? Can those tests requiring UI interaction be separated somehow?

Following your suggestion I ran tests for Testcodeunit 134006 manually and they all passed. Doing the same via consolemode fails. It appears the shown progress bar might be responsible.

Beside those failing tests I ran into the following exceptions during my build: Error3.txt Error1.txt Error2.txt

I'm not sure if asking you to look into them is appropriate at all.

[tfenster]

The error messages look different than I would have expected but I am not sure how much time it makes sense for you to spend there as indeed your statement is true that test are not designed to run without a client, so a lot of them will fail

[andreasblueher]

Hey Tobias, I continue to find blog posts and talk to people on this topic and it appears it should be possible. One major finding we had today was the container needs 8GB of ram to work properly. Now the crazy error messages seem to be gone and we're getting more expected messages like: "Customer ledger entry not found" which is not completely expected, but much closer to a test result than before.

Looking at this blog or this YouTube Video from Jasminka automated test execution should be viable.

So right now our test result output from Azure DevOps matches the output from Test Tool in the client (from the looks of it). I'll keep playing with different docker images to find out how test results differ. If you're interested I'll keep you updated and thank you for looking into this.

[tfenster]

Well, the blog post explains the problem about not being able to use a clientless configuration. The "solution" is to start the RTC headless but that won't work in a container as well. At least it didn't when I last tried.

I've seen some indications that MSFT will maybe release some kind of automation API which will allow true clientless testing and I would guess that the latest point where this has to come is when RTC is deprecated but until then I am quite sure that you will need an RTC (headless or not) and that breaks the scenario to run it in a container. Maybe the new "windows" base image could be an option as it should have some headless GUI capabilities, but I haven't tried that yet

[andreasblueher]

Let's hope Microsoft improves the experience and tool chain even further, but for now I can report it is possible to run the tests via Azure DevOps inside a container using the consolemode to start dynamicsnav client.

The 152 failed tests we got indicate indeed errors which need to be fixed. Also there are some "Unhandled UI" errors, but they don't shut down the pipeline as I would have expected, only the test fails. We're currently using the bcsandbox:base image.

[tfenster]

@andreasblueher wow, nice! Thanks for letting me know. Is that on a Server 2016 or 2019?

[andreasblueher]

It's still the 2016 server, but we're planning on giving a 2019 server a try soon. Will keep you updated.

[tfenster]

I can't find my old stuff anymore where almost everything failed. I'd be very interested to see if it indeed is something changed in NAV / BC