MAGN-7252 As a developer, I want to run Revit test cases of one assembly in one session, so that they run in less time.

[Randy-Ma]

Purpose

When running test cases continuously, it may get issues like: 1). There may be one test case which cause a crash; 2). There may be one test case which introduces an infinite loop; 3). Upon running one test case, the memory consumption may be too high; (The 3rd case is not addressed in this pull request here because right now the memory consumption is acceptable which will not block the test cases)

For these cases, originally, the test application won't know the situation and does not know how to solve them. Thus if one test case has one of the mentioned issue, the result of the whole test cases will be impacted.

In this implementation, we will try to introduce a mechanism to monitor the running process through socket communication. The test application will keep an eye on what's going on in the main application. If the main application hangs, it will kill the process and record the culprit and continue running the remaining test cases.

Implementation

In the runner, a server will be constructed to listen to reports which may be sent back from the main application. In the RTFRevit addin, a client side will be constructed which will send message to the server.

The original journal file for all test cases will be prepended with an RTFClientStartCmd which will pass port number to the client through journaling data. An RTFClientEndCmd will be appended to identify that the client has completed running test cases. These two new commands will be implemented in the RTFRevit addin.

A class Message is defined to be passed from client to server. This class is used to contain data like status information, or test case information to be passed. It has two derived classes: DataMessage and ControlMessage. DataMessage is used to pass test case information and ControlMessage is used to pass status information. The Message classes can be serialized into and deserialized from byte arrays.

Each message has a header to identify the length of the message. Each message will also has a ID which will increase automatically when a new message is created. The length in the header will first be read and the corresponding count of bytes will be attempted to be converted to a message. If the conversion failed or the message ID has not increased by one. This means there are something wrong. (The bytes will be assumed to be little-endian)

Test

A sample test assembly has been implemented which has one test case to simulate a crash and one to simulate an infinite loop. Here is the test result:

The two which have errors are identified and we can see from the text output the journaling file has been recreated between Test4_Infiniteloop and Test5.

Reviewers

@ikeough

[ikeough]

@Randy-Ma Wow! Let me see if I understand...

• RevitTestServer is a socket server that is started and ended by the Runner.
• Port information will be conveyed to Revit via the journal file, so communication can be established with the RevitTestServer.
• RTF will register two additional IExternalCommands, RTFClientStartCmd and RTFClientEndCmd with Revit.
• The RTF client on Revit will send a message every time a test starts. This way we know what test started. So, if a timeout is hit, we can mark that test as a failure and restart the system from the last failed test.

Is the above understanding correct?

I think the server idea is cool. But, let me ask a dumb question. As each test runs, the test results are written to the .xml results file. That results file represents the record of all tests completed so far. So, if you get into any of the three situations above, and you need to abort the test by way of bringing the whole system down, can't you just read the results file, and see what that last successfully completed test was, and restart from the next test? What additional benefit does the server give us?

[Randy-Ma]

@ikeough Yes, your understanding to the implementation is correct. The reason to set up the server is that we need a way to know the mentioned three cases are happening. When running continuously, the gui/console application can not differentiate whether the main application is normally running other test cases or is hanging/crashing. But your question reminds me of another way by possibly keeping watch on the result file, if the file does not change for a while, we assume the last running test case is having some issues and close the application and regenerate the journaling file and run the remaining test cases in a new process. This way I think is also feasible if we ensure after each test case is run, the result will be immediately written to the result file. Right now, it looks like one TestThread can run a fixture and update the results together, in this case, we might need to change this a bit to update the result separately for each test case to use the xml file for information exchange.

[Randy-Ma]

I am right now sticking to the IPC approach using socket communication because at least I have not seen drawbacks so far. One advantage is that since the test case to run is known, it is easier to identify which test case hangs. It is also possible to visualize the running process so that users know which test case is running after which test case.

[Randy-Ma]

@ikeough The code has been updated and also I have updated the summary. Please take a look.

[ikeough]

@Randy-Ma Have you considered using WCF? I got the feeling reading your socket server code that there HAD to be a library out there that made this easier. It's up to you. If you google "sockets versus WCF" you'll get a lot of S.O. posts debating the merits of each. If WCF sacrifices speed in the interest of having more readable code that doesn't have to deal with checking byte arrays for message lengths, than it would be preferable.

If you are interested in sticking with the sockets approach, then I just have the one small comment.

[Randy-Ma]

At the very beginning, we have had a discussion about using WCF or sockets or other technologies. I know WCF before but my feeling is that if you are not really familiar with it, it may take you time to solve some issues that may occur in the process. Unlike sockets, the concepts are easier to understand, at least for me. But like you have said, I may not have to deal with byte arrays and message lengths if I am using WCF. What is done here is like implementing a small protocol - maybe it is not very proper to call it a protocol.

I prefer to take the sockets approach for now while doing some investigation to use WCF and possibly some comparison between the two in the future.

[ikeough]

@Randy-Ma Thanks for the explanation. Let's get this in and running, and we can re-open that discussion if/when we decide to add more complex messages.

LGTM.

[sharadkjaiswal]

Great stuff @Randy-Ma

[Randy-Ma]

@ikeough @sharadkjaiswal Thanks for the comments! They are very helpful!

[RodRecker]

@Randy-Ma nice work with this. When will we turn it on in the CI system?

[Randy-Ma]

I will discuss with @riteshchandawar about this and hopefully it can benefit us earlier.