bluebird75 / luaunit

LuaUnit is a popular unit-testing framework for Lua, with an interface typical of xUnit libraries (Python unittest, Junit, NUnit, ...). It supports several output formats (Text, TAP, JUnit, ...) to be used directly or work with Continuous Integration platforms (Jenkins, Maven, ...).
Other
565 stars 136 forks source link

Inaccurate Time #140

Closed luainin closed 2 years ago

luainin commented 2 years ago

Thank you for writing luaunit -- it's very simple to use.

It seems that the timings for the test runs are inaccurate -- I believe this is due to the usage of os.clock() which measures only Lua engine cpu time, as opposed to "real time" (at least in my experience). Unfortunately, I have code that makes http calls (yes, these are not "pure" unit tests), which makes calls to C functions and thus, do not factor into the "total" time for a test.

Any suggestions on how to go about this issue? I replaced all appropriate os.clock() invocations with os.time(os.date("!*t")). This gave much more accurate results; however, the precision was only to the second.

bluebird75 commented 2 years ago

Sorry, I totally missed your report.

In lua, we are limited to what the lua api provides. As you found out, it's one of : time(), date(), clock() . See https://www.lua.org/manual/5.4/manual.html#6.9

Using os.clock() was a mistake now that I reread the documentation. When choosing it, I was happy because it was the only one with sub-second precision but I missed the fact that it is not measuring time but CPU-time. So, not the right choice for my goal.

The other functions do not go beyound second precision, which is a pity. In theory, time() returns a double which would be fine for us.

I'll try to look more into the topic but with stock lua, there is not much more we can do.

bluebird75 commented 2 years ago

Threre are basically two solution to your problem. It's been a long time so I guessed you figured it out already but still.

Solution 1 : depend on an external package providing a more precise time function. Many external lua librarires provide this, See different solutions proposed in : https://stackoverflow.com/questions/463101/lua-current-time-in-milliseconds/66402619#66402619

However, I want luaunit to stay standalone.

So, I'll go with solution 2 : use os.popen to ask the shell for a more precise time. I'll go with this for a future version of luaunit. And I will also state this clearly in the documentation, and provide an easy way to replace the timing function().

And in the meantime, I will change os.clock() to os.time()

Thanks for your report.

denisdemaisbr commented 2 years ago

maybe

https://stackoverflow.com/questions/463101/lua-current-time-in-milliseconds

luainin commented 2 years ago

@bluebird75 Thanks so much for your response:) Yeah, I had ended up making a copy of luaunit and made a few custom additions to the library to fit our needs, among which was the time modification.

Sorry for not closing the issue sooner -- I had forgotten all about it since this was a simple fix on our end:)