Order of test function calls differs when run on (M1) Mac vs. Linux

[krispin-gradient]

I'm running unit tests locally on a Mac, for code submitted to a cloud test process running on Linux, and noticing that the order of function calls differs locally from remotely (see case below).

Is this an expected limitation of Ceedling? Is there any way to address it (beyond setting enforce_strict_ordering to false?)

Any advice would be greatly appreciated!

Example Case

Given a function of the following form:

int funcA(void *context) {
    funcB(funcC(), funcD());
}

The local test for this function passes when it is ordered as follows:

...
    funcC_ExpectAndReturn(x);
    funcD_ExpectAndReturn(y);
    funcB_Expect(x, y);
...

The remote test for this function passes when it is ordered oppositely:

...
    funcD_ExpectAndReturn(y);
    funcC_ExpectAndReturn(x);
    funcB_Expect(x, y);
...

What to do?

[mvandervoord]

Hi.

You're running into undefined behavior of the C standard. C does not guarantee an order it will evaluate the arguments passed to a function. Therefore, if a compiler decides it's more efficient to run D before C, then it can do so.

A couple things you could try:

(1) disable optimizations for tests. This MIGHT make them default to the natural order (still no guarantee). (2) Call funcC on its own line and collect the result. Do the same to funcD, then pass the results. You'll have forced the order. This (of course) assumes you can alter the release code. (3) As you mentioned, disable global ordering.

As of right now, CMock does not have a feature to enable and disable the ordering during a test, or to otherwise mix and match. Sometimes there are workarounds, though.

(4) For example, if my test looked like the following, it would always pass:

...
    funcD_IgnoreAndReturn(y);
    funcC_IgnoreAndReturn(x);
    funcB_Expect(x, y);
...

In this case, it isn't EXPLICITLY checking that C or D get called, but allows them to be called 0+ times. If they DO get called, it specifies what they should return. So we are verifying that C and D were called at least once each and the values were correctly assigned to the B call.

In many cases, the above is sufficient, and often it's better because it focuses on the important behaviors.

Best of luck!

[Letme]

I would add some more information (option 2 from Mark's post) here. If you want order of variables kept, then please by all means assign the return values of functions to variables before calling a new function with them. That way they also end up on stack. But as mentioned by Mark, you will lose some compiler optimization options (you can consider compiler smart in these usual cases), but will get much more predictable code.

int funcA(void *context) {
    int varC = funcC();
    int varD = funcD();
    funcB(varC, varD);
}

[krispin-gradient]

@mvandervoord, @Letme, really appreciate the explanation. Thanks!