How can I check test result in an async callback

[myzhan]

Hi, I'm using luaunit to write unit tests for my android app. Think about I write a HTTP module, which send requests asynchronously by Java and handle responses in a lua callback function.

And I write a test like this.

function testGet()
    http.get("google", function(response)
        if response.status_code ~= 200 then
            luaunit.fail("status code should be 200")
        end
    end)
end
runner:runSuite()

The test suite ends before the callback is called. So the test result is not collected.

Any idea about how to write test cases in this situation?

[bluebird75]

Let me try to understand this.

When you call luaunit.fail(), regular lua error is generated. What probably happens is that the http library which you are using is managing the error and somehow ignoring it.

A point of investigation is this http library : can you detect when an error has been triggered in the callback function ? If so, this is probably the right path to verify your functions.

[myzhan]

When http.get is called. It submits an async task to another thread, then it returns without errors. So runner:runSuite reports that all the tests is passed. But later, we get a http response and check the result in the callback funtion.

Can I ask runner to wait until the async task is done?

[bluebird75]

Running a unit-test is a very synchronous task. When you are finished running one test, you want to have the result of the test. Adapting luaunit for an asynchronous usage would be an interesting challenge but I don't think it will happen soon.

With that said, you have to make your test synchronous. This means, inside the test body, wait until the http.get is completed and assert that no error were raised. Do you have the equivalent of thread.join() for your http request ?

In luaunit, the assertion verification is based on triggering a lua error and catching it. In the asynchronous call, the error is raised but the test body does not catch it. So you have to find a way to propagate that error back to the test body. This might mean using a dedicated variable for this, and benefitting less from luaunit's automation.

I guess something along the lines of :

function testGet()
    done = false
    error_msg = nil    -- error_msg being not nil means an error is to be reported
    http.get("google", function(response)
        if response.status_code ~= 200 then
            error_msg = "status code should be 200"
        end
    done = true
    end)
   -- thread.join() or wait until done is true
   luaunit.asserNil( error_msg )
end
runner:runSuite()

[myzhan]

Yeah, I'm thinking about something like that. But currently in my implementation, only one thread can hold the global lua state, so runner:runSuite must return, or the callback function won't be called.

Maybe I can collect the test results in callback functions without luaunit if there are not better options.

[bluebird75]

I believe we can do better, but I do not have time right now to experiment with the approach.

If you look LuaUnit.execOneFunction(), you can see that the status of a test is updated by calling addStatus() (which could be renamed to updateStatus()). addStatus() knows which node to add status to by using self.result.currentNode.

You could refactor addStatus to have addStatusToNode( node, err ) so that you are able to pass the node explicitly. This gives you the ability to update the status of a test asynchronously, as long as you keep a reference to your node inside your callback (meaning, you create a closure).

The next problem is that endTest() is used to update the global test count, so either you have to fix it when you update your status, or you need to find a way to asynchronously call endTest(). The same holds for tearDown(), endClass() and endSuite().

The next problem is that this will completely mess up the display, as test output relies on the fact that tests start and finish one after the other.

What is probably needed is a complete asynchronous mode, where the async callbacks would be responsible to inform the framework that a given test is completed. It is not that difficult to turn LuaUnit into this, but it is certainly not ready now.

[myzhan]

Thanks for the detailing. I will find some time and look into it.

[bluebird75]

I will give it a try in the next few weeks, the challenge of handling this has resurrected my motivation !

[bluebird75]

Which lua http library are you using ? I would like to validate my code with it...

[myzhan]

Oops, I'm not using a pure lua http library. We use JNI to write a lua module named http, which will eventually use a java http library.

I can provide a minimum android app with source code for you to test. It's that ok for you?

[bluebird75]

Ok. Don't bother with a complex project. I'll just use some simple async library to build my tests, that should be sufficient. Even coroutine might fit the minimal bill for verifying that luaunit works with async code.

[myzhan]

@bluebird75 Hi, any progress on the feature request?

[bluebird75]

Finding an async library turned out to be more difficult than I thought. I really need to install a full blown thread management library to reproduce the async test case and I have not done that yet. I still want to do it but need to find motivation again :)

[myzhan]

We really need this and I can help you to test.

[bluebird75]

I need to understand your situation slightly better.

You only have one lua state right ? Have your callback function access to that lua global state ? Meaning, can your lua callback function read and modify variables of your lua state ?

The second thing I understand is that when you call runSuite(), your second thread is actually not executing and you have to wait until runSuite() returns to start executing your second thread and get its result ? If that's really the case, that's a blocker. You need to find a way for luaunit to sleep and let the other thread do its execution until it is completed. The traditional way of doing this with threads is to call join(), or to sleep() until a given condition is occuring, letting the other thread do its job.

[myzhan]

1. The first question, yes. I have only one lua state and the callback function have access to it.
2. Yes, runSuite returns before the callbacks are called. I'm thinking of keeping a counter of running async tests and separating runSuite into startSuite and endSuite. When startSuite returns, I can start to poll the async test results periodically. If all aysnc tests end or timeout, I can call endSuite to collect test results.

[myzhan]

Hi @bluebird75 , I find a way to mock the http module and make it synchronous. The "async mode" is not so necessary now. Thanks for your help anyway.

[nicolasnoble]

So, I can see this issue has been closed as completed 4 years ago, but I don't really see anything in the documentation explaining how async testing works at the moment. I've been trying to trick using coroutine yielding / resuminng, but I am getting some strange results so far, that I have yet to debug.

Was there some actual feature added to support async testing?

[nicolasnoble]

Ah, I take that back. I just managed to get it to work using coroutines properly. I may send a PR to make what I'm doing a bit more palatable.