Automation / Testing / Visual documentation framework

[ocornut]

(EDIT 2019: Even though this is an active topic (in 2019) note that this thread started a looong time ago and the early messages are not representive of where we are going now. When the new testing framework is a mature enough to be shared I'll close this topic and open a new one.)

(EDIT 2022: Released Dear ImGui Test Engine + Dear ImGui Test Suite https://github.com/ocornut/imgui_test_engine)

Looking for someone who would be interested in working on that, probably under my guidance.

Mentioned it in #259

Add a framework for generating screenshots/animations for given pieces of code. This is probably going to be done along with the "testing" framework. I would like the ImGui documentation to be a very visual, well organized listing with C++ code on one side, screenshot (or animated GIF) on the other side and tags + commentary. I think this will easily convey a lot of ideas and tips about ways to use ImGui. Update: I've created some helper to help me take manual screenshots and its been very helpful in the past few days. This will probably evolve into such framework.

The idea would be have a separate test application, could be or not on the same repo, that would A/ generate screenshots/gif + code markup output for the wiki and B/ defacto run tests.

So we could describe those documentation elements in the code, the framework would execute the test, simulate inputs to interact, take screenshots or gif, crop them as desired, parse its own .cpp file to retrieve the code (which itself could have marking in comments to hide some lines) and generate the help section that would demonstrate ways to use ImGui by showing the code and the screenshot. We would have hundreds of short examples, grouped, chaptered, tagged, indexed.

I was originally primarily interested in the value of documentation but I think the overlap with testing will be enormous and natural. Just the act of generating the documentation from runtime will be an act of testing. The testing wouldn't be unit tests at ultra fine granularity but rather functional tests (at least this is what I think they are called in the real world?): say, move mouse to corner, click and drag. We can compare screenshot, regression easily caught. Tests would be merely the unpublished part of the documentation system. Any tricky area of the code or fixed bug can be subject to a test, etc.

It's not really a trivial task to set all that up and I think it'd be highly valuable to the project. Realistically I don't think I'd be able to tackle by myself this year, so looking around if someone is interested to contribute :)

The person would be expected to be proficient in writing terse, optimal C++ and willing to adopt the general programming style of ImGui.

Thanks!

[ocornut]

PS: to clarify even if we could aim to have lots of it in a separate file, that would still involve touching a lot of the core library, and some of the changes will probably lead to imgui's own public api development.

[emoon]

This sound like an awesome idea. I have thought about this before as I want to document the UI API in my debugger. The API is close the exactly the same but with a different naming scheme but that is very minor. I would likely try to use this as well (given permission of course) in a slightly altered form.

[emoon]

I'm bit interested in helping out with this.

I wonder if you have some ideas on how to do it? That being said I don't have lots of time doing this but I want to help out at least.

[ocornut]

Thanks Daniel and sorry for not answering to this earlier. :) Some out-of-order notes on top of my head.

That picture is essentially showing the intent:

Except it's all wrong because:

• the code is embedded in a picture
• it's just showing random unrelated stuff
• the example isn't complete and lack context
• and mostly, it was painfully and manually authored

I think a first step would be to manually draft more example documentation. Which would naturally lead to creating the framework for this. It would probably embed something like the screenshot helper here https://github.com/ocornut/imgui/wiki/screenshot_tool

How should the documentation be presented? When and how can we be skip context? Then we figure step by step out how we can make this work easier and more automated. Believe it or not I feel pretty bad with my English when it comes to presenting things in a consistent manner so I could do with some help :)

I am not too concerned about tackling the testing part too explicitly at first. ImGui doesn't really crash so tackling this possibility in an automated manner isn't an important priority ihmo. I think for a start the documentation itself can be defacto testing.

Overview style vs Single-function/symbol style

I think there would be more value in providing chapters/article about series of functions rather than individual functions. There's nothing more frustrating that documentation which have a line about each functions but you never get to understand the whole picture. So I would rather have a chapters called "Menus & Popups" and say, BeginMenu() EndMenu() are documented together, than separate entries. It's still possibly to have separate entries for every functions as the second half of those chapters if we feel it makes sense. As the output will be automated we will be able to build a full index anyway.

Also, the documentation shall be seen as a general "usage guide" which include suggestions, tips, making uses of several functions to benefit from a certain pattern, etc. so I feel it is nice to not overly focus on single function/symbols.

Note about repo size: I would prefer to keep the main repo small because any commit ends up in user clone. So if we end up uploading pictures they could be on the Wiki repo (which would make sense for the documentation anyway) or we create a imgui-docs repo just to host the data

[ocornut]

Here's a quick example I just hacked EDIT and here's the code: https://gist.github.com/ocornut/4d9001336d07a6818f96 Code is pretty bad/hacky/shortsighted but just giving the general idea of where things might go. It only generate the pictures for now. It should eventually generate a full .md file with text and links to screenshots?

---

Buttons

bool Button(const char* label, const ImVec2& size = ImVec2(0,0))

Button() returns true when it has been pressed

    if (ImGui::Button("Save"))
        MySaveCode();

Button() can takes an optional size, set individually for each axis. A value of zero sets the size automatically based on the label.

    ImGui::Button("Click");
    ImGui::Button("Click", ImVec2(0, 0));

A positive value sets the size

    ImGui::Button("Click", ImVec2(100, 0));   // Specify width
    ImGui::Button("Click", ImVec2(0, 100));   // Specify height
    ImGui::Button("Click", ImVec2(100, 100)); // Specify width and height

A negative value align to the edge of the window or current contents region

    ImGui::Button("Click", ImVec2(-1, 0));    // Align to right edge
    ImGui::Button("Click", ImVec2(-20, 0));   // Align to right edge 
    ImGui::Button("Click", ImVec2(0, -1));    // Align to bottom edge 

---

[emoon]

This sounds good.

I also would like to separate the code examples into separate files that may or may not be included when generating the doc. That way it's possible to have allow others to use the documentation but having slightly different syntax (for example in Rust or C wrapper case) but still use the guts of the documentation.

Still the generated doc can include samples with the code in question but selectable in some way.

Also using markdown for the documentation would be great.

[ocornut]

Yes ideally it would become a set of helpers that can be used in different contexts, locally or for languages ports. It could potentially even be useful for people to document their own imgui-based helpers or even post repros here :) but we'd have to see how it evolve. Shorter term I would just suggest tackling the basic problems: parsing source code, outputting to markdown, adapting the system so that it works in more contexts. Some context may include or exclude the creation of a window, some context may need to hide some code, etc. I could create a imgui-doc repo right now and grant you access if you want to fiddle with it together and we can always scrap the repo if it ends up back inside the main repo later

[emoon]

I was thinking of doing it the other way around.

Parsing markdown (with inline or ref code) and generating code to be executed and then screenshoted. That way one doesn't need to parse source code as markdown is likely way easier to parse.

Thoughts?

[ocornut]

I would guess it is easy to parse source code as long as we have known markers and don't try to support edge cases, and it's nice to write and iterate/run actual code directly in place, rather than indirectly via a MD layer. But either way would work.

I semi expect the code to include various markers/oddities. (e.g. a marker to request "hiding a line" or hiding unnecessary scaffolding from from the user-visible output) so even if MD is used as source, we would have to regenerate a different MD either way. For example for a particular series of example demo we may need to setup certain conditions but maybe the setup is only visible once (or never if it's not really important in context).

[emoon]

Yeah if we use markers then it should be fine.

[ocornut]

Created an empty repo for code https://github.com/ocornut/imgui-doc I don't mind keeping the discussions here as the other repo may or not last

[emoon]

Thanks! I will give this some more though and come up with a suggestion.

[emoon]

Sorry for the lack of updates on this one. I will try to get it done this week.

[ocornut]

No worry! Been myself going to a crazy patch of work from which I hope to escape soon :) Thanks for looking into that.

[emoon]

Still haven't got around to do anything more about this :( If anyone else wants to help out with that would be great but my plan is still to look into it even if it will likely ta a while longer than I hope.

[RobertoMalatesta]

@emoon see my comments on #693 I'd appreciate your opinion.

--R

[RobertoMalatesta]

Hi @ocornut , @emoon, it's not clear to me (and maybe only me) what the depicted system will try to achieve. It seems great, fantastic. On paper.

I semi expect the code to include various markers/oddities. (e.g. a marker to request "hiding a line" or hiding

How badly having new macros or directives will impact the terseness & understandability of code?

And, first and foremost, which ETA? (you know, I'm as curios as a cat, and I'd like to be still alive, the day this systems comes out)

Set aside the automatic testing issue (up until someone eventually solves the Halting Problem), I wonder when this system will be available.

Wouldn't be more realistic --that's a nice name alternative for IMGUI: RealisticUI btw!-- to use already existing and not-too-complex-to-learn tools ?

I will state my proposal in a concrete way:

1. adopt DoxyGen as a simple, unobtrusive way to document and map code (up to graphs and UML charts)
2. implement an official emscripten port of imgui (I would offer mine, but it's two years old and @floooh one is definitely the best.
3. create an official way and directory to create extensions and plugins for IMGUI and document it as a priority issue.
4. drop some basic examples there along with some templates to creating new ones
5. create a script to build and generate a page containing source code on the left and running, live example on the right using 2.
6. link doxygen autogenerated documentations to examples at point 4 (automatically, with tags inserted in normal source comments)

That's basically all. No new macros that will turn IMGUI into BOOST, no new system to create falling victim of NIH Syndrome.

--R

[ocornut]

Doxygen is just not good enough and won't provide a testing infrastructure. Testing is the first need. There's no ETA, imgui is a hobbyist project. The aim is to make something great nothing less. If we were low-aim realists we'd be stuck with QT forever. I may have more time for ImGui from next January. It will also be an ever increasing importance to many big ImGui users (game studios) that those testing systems get implemented so perhaps they will have more incentive to help.

How badly having new macros or directives will impact the terseness & understandability of code?

Those would be in the testing code, not in core imgui.

Your proposal doesn't solve the testing requirements, and pardon me but doxygen is quite obtrusive. Many imgui users would be freaked out if we doxygened the code. This library is built in such a weird/specific unorthodox way that classic doxygen just isn't so appropriate.

Right now I think the documentation isn't that bad for experienced programmers. As strange as it seems, it is not the current priority to make imgui accessible to less experienced programmers. Just because we don't have the bandwidth/resources to handle influx of users and solve every problems. It needs to mature slowly. And I think the library isn't finished. It may take one year, two years until it feels finished. You are right on many points, we should improve extensibility, improve examples, but that's all lots of work it doesn't happen magically. Right now I think the focus should be testing, adding useful features, improve code sharing, attract experienced programmers who can help. Accessibility will slowly improve :) I just only have so many free hours, need to make choices. Sensible contributions from imgui users are most often welcome.

[RobertoMalatesta]

HI Omar, and thanks for the time taken for giving me a detailed answer.

doxygen is quite obtrusive. Many imgui users would be freaked out if we doxygened the code.

I'm not trying to sell you something, so it's OK for me.

Testing is the first need.

one thing I left out was:

7 . implementing a web-based playground where any developer could fiddle in code and experiments: a small app template inside a page, with a compile and run option. That way proposing examples and components or signaling bugs would be easier and done with a standard approach.

Right now I think the documentation isn't that bad for experienced programmers.

Agreed, but if something takes me T instead 10*T I prefer the first route nevertheless. And documentation is more than a quick route for n00bs, it helps define and formalize the API pact between who's using it and the developer of the component.

we should improve extensibility,

Really. More than one year ago I implemented a basic chart wiget to do pies and histograms, along with a calendar control. It was a day's hobbyist work as a test of what IMGUI could give me. The code was cutty-pastish, but it worked. Then I thought for a while how to clean them and how to do a push on github. But everyday's work took over and those components went forgotten onto one disk somewhere. Pity since those snippets could have been useful someone else to start over with.

In different github repos I see interesting components developed on modified forks. A pity that the lack of a common extension system does not allow us to summon them all. This should be priority one,

--R

[dougbinks]

I'm putting some thoughts on this issue here after a brief discussion on twitter:

So there are two major parts to this, an example code document generation framework and a regression testing framework. The idea is to make these work together so the examples are also tests, making it easier to maintain.

Documentation Framework

The documentation system seems doable in a fairly straight forward way using standard C++ and macros. I think the requirements would be something like:

1. Minimally intrusive to example code.
2. Not required to be present in a normal build of the examples.
3. Easy to use.
4. Lightweight, pref single .h/.cpp pair.
5. Automated.

For 1 I would use something like this example but with IMDOC_BEGIN not creating a window as I think this obfuscates the original source code. So I would have something like:

IMDOC_BEGIN("Button: Default Size");
IMDOC("Button() can takes an optional size, set individually for each axis.");
IMDOC("A value of zero sets the size automatically based on the label.");

    ImGui::Begin("Button: Default Size");
    ImGui::Button("Click");
    ImGui::Button("Click", ImVec2(0, 0));
    ImGui::End();

IMDOC_END();

Note that this approach makes the ImGui::* code able to be pasted into code not using the doc system, and more easily searchable.

Requirement 2 is simple enough using macros, as the regular imgui include can define the macros as producing nothing.

To be easy to use means that the output should be a .md file and a directory with the required images in it. Preferably that file would contain all the documentation, so we may need macros to be able to include source .mds for long preamble you don't want in the .cpp file.

The lightweight part is debatable, as leveraging scripting tools or lang tooling might be more powerful, however it's doable and probably faster to get the basics up and working.

To be automated should mean that simply running the code gives the output with no need for user input. This would mean no default closed windows, or that the doc system can control whether a window is open or not.

Regression testing Framework

This is where things get tricky.

I think I understand the objective - inject input and capture screenshots (we may want to capture the entire back buffer to ensure we capture changes in imgui window position) then compare against prior data and flag changes to some easily viewable output.

The following code is for a demo which doesn't open new windows and where we want to test widgets in turn (range based, euto etc for brevity):

test_setdrawfunc( demo );
auto windows = test_get_windows(); // runs demo function, gets list of windows

for( auto window : windows )
{    
    auto widgets = test_get_widgets( window ); // see below
    for( auto widget : widgets )
    {
        test_hover( widget );
        test_click( widget );
        test_doubleclick( widget );
        test_entertext( widget, "foobar 99.0 []&$" );
        test_clickdrag( widget, 0.1f, 0.0f ); // units in size of widget
        test_clickdrag( widget, 1.5f, 0.0f );
    }

    test_drag( window );
    test_move( window );
    test_size( window );
    test_close( window );
}

test_get_widgets( window ) would work by iterating the mouse coords over a window and getting all different HoveredId's from the ImGuiContext.

Each test function would likely need to reset state, run the demo function for a few frames, add the relevant input whilst running the demo and capture & output pngs and any required data.

Code would then need to be written to compare pngs and generate a table of results.

[RobertoMalatesta]

I'm still skeptical on this but I'm willing to give full support to your effort. @ocornut is our Obergruppenführer since it worked from day one to now- --R

[ocornut]

In spite of some of the kind offers above to help, I haven't got to give time to this topic. My apologies Doug! I think in 2018 one of my biggest hope is to build this. At the time I was mostly focused on the visual documentation part, today I would move the balance a little more toward the testing part (but both can still benefit).

Many of the useful tests we can perform won't need to involve any rendering. If we are going to test things like "press arrow down to move selection lower" we can compare selected/hovered/active ID, or the effect is has on your user side state. Visual comparison can complement the testing but not every test needs it.

I expect our testing framework to have helper like "locate ID "xxxx" on the screen". The work leading to the navigation features that sort of things easier to achieve.

Rendering wise, some things to ponder: ImGui can runs without a renderer, a lot of the testing can happen without OpenGL etc. involved. ImGui emits vertices, so if we need to do visual comparing, perhaps comparing vertices directly is better. It's won't be prone to renderer differences, and we even have a little bit of extra information at the ImDrawList level. ImDrawData can also be rendered using a software renderer. If that's any useful, we can imagine a testing framework that emit binary render data blob storing ImDrawData and have a renderer for that. Anything is possible here, if we deem it useful.

In my mind, testing would consist of:

• A "gui function" that emit regular imgui output. This may be shared by different tests.
• A "test function" that manipulate and test the widgets, would be written in a sequential way although it would run interleaved with the first Occasionally the GUI function may have test-specific code if required.

// This can become more elaborate, we can have big GuiFunctions that are used for dozens/hundreds of tests?
void GuiFunc()
{
   ImGui::Button("Hello");
}

// t = time in frame
void TestFunc(int t)
{
    // Script triggering a mouse move over 10 frames. The test engine will locate "Hello" (that will take 1 frame) then move the mouse.
   if (t == 0) MoveMouseTo("Hello", 10);
   if (t == 100) CHECK(IsHoveredByName("Hello"));
}

This is not a full fledged idea at all, I think we can only clarify this idea by putting it to practice. Interestingly we can run those tests at light speed, they don't need to be run at 60 FPS and they don't need to be rendered. Only if I test fails we can go into "slow / interactive / give me a render" mode.

[gplusplus314]

Are you proposing that GuiFunc() is responsible for essentially "scaffolding" out a GUI, then TestFunc(...) performs operations and assertions on the scaffolded GUI?

In concept, I'm thinking about three areas of functionality:

1. Test fixtures/scaffolding - code that creates generic GUIs/widgets/"things" to be tested. Has no knowledge of specific tests or how to automate the GUI.
2. "Driver" code to interact with a GUI ("find widget by text" and "click widget" kind of stuff). Has no knowledge of specific tests or how to create the GUI.
3. Test-specific code that consumes points 1 and 2, but does not actually use Dear ImGui directly.

Within reason, this pattern allows for breaking changes in the API to be isolated, rather than affecting the tests themselves. Also, it would make it very clear as to which parts of the code are safely reusable among individual tests.

Just thinking out loud here. What are your thoughts?

Edit: so basically, point 1 would be where we take care of the documentation and screenshots, as previously discussed in the thread. I suppose a drawback is that we wouldn't be reusing the existing examples. Hm....

[ocornut]

Are you proposing that GuiFunc() is responsible for essentially "scaffolding" out a GUI, then TestFunc(...) performs operations and assertions on the scaffolded GUI?

Yes.

My thought is that I don't see why 2 and 3 are separate things, we'll get smaller and simpler code merging both (as in my example). Many tests will be super short and I feel that interacting back and forth between 2 and 3 will cost us more overhead than benefits. I don't really mind any specific architecture tho, I think it's best to just go and try to implement 20+ tests before we can narrow how we deal with them.

Part 1 will be the more reusable and we can feed them with custom flags/parameters so it would become acceptable to have big GUI functions along with very small GUI functions. In fact the ShowDemoWindow() might even be an acceptable gui function to be used by some tests.

Not sure when exactly I'll be able to focus on this yet. I had so many regression with the nav branch and docking that I feel it would be useful to address that sooner. I'll see if I can secure a week+ from my current tasks on viewports/docking to iterate on a first version.

I also don't know if there is a point in using a unit-test framework, e.g. https://github.com/catchorg/Catch2

[gplusplus314]

I completely agree with writing 20+ tests first, then figuring out the abstractions as pain points are discovered.

My thoughts on separating 2 and 3 is that, for the most part, almost every test will have to MoveMouseTo(...) and other similar things like Click(), so I don't think it should be re-implemented in each test.

No opinion here on using a test framework. I'd gravitate toward starting with just a quick-and-dirty approach on a separate branch first (or the imgui-doc repo), then make an assessment?

Edit: Looking closer at your example, I think we may actually be saying the same thing in a different way. You said the "test engine will locate 'Hello'" - makes sense to me.

[floooh]

FWIW (since I was summoned in a comment above): I'm completely happy with the current state of documentation (inline comments in the header, and a demo program which tests/demonstrates all features) :) What I usually do when trying to achieve something is first find something similar from looking at the demo, and then find the demo source code which implements this. I think having an 'explorable' interactive demo for this is very important (vs screenshots in a separate static documentation).

The only feature I can think of which would improve the current state would be a 'help mode' in the demo. For instance press F1, and the next click on an UI element in the demo takes me to the source code line on github where this UI is created in the code (most of the time I'm looking at the demo source code, and only go to the header from there if I need more info).

[RobertoMalatesta]

Funny thing. Some years ago I published a message containing the internal URL to my Doxygen IMGUI page: IMGUI 1.50 WIP Doxygen -- outdated

I pulled it out some month ago since I'm not using IMGUI now.

Then I noticed a raise in 404 errors given not only by search engines but by a fair amount of ppl that were indeed using it, so I restored it from a backup.

Month by month the number of unique IPs using it is constantly increasing, so someone is finding it useful. -- (with time I will update it to 1.60 as soon it is final).

Figure: (500+ different IPs per month doing search calls -- not all of them from search bots. Even 1/10 pristine users would be significant).

The only feature I can think of which would improve the current state would be a 'help mode' in the demo. For instance press F1, and the next click on an UI element in the demo takes me to the source code line on github where this UI is created in the code (most of the time I'm looking at the demo source code, and only go to the header from there if I need more info).

Doxygen can produce XML code after the make from IMGUI source code as it is now, then this XML can be filtered to feed a C++ structure for an IMGUI component sporting a classical class/function/const inspector tree and a quick search field.

Size would be bigger than the demo, but this compo could be used as a walk-through intro from documentations to pop-out examples+source, maybe later extending it with a small tutorial (I agree: no need of it, but the easier the quicker).

No code attached, sorry, but at the moment I'm not using IMGUI in my projects.

--Rob

DISCLAIMER: the author of this post holds no stocks or interest related to Doxygen.

[gplusplus314]

Sorry to bring back an old topic, but I think I may have identified my mental disconnect between what we're talking about here and what was going on in my head. I'm bringing this up so that maybe, if someone else was thinking the same thing, we could differentiate between the two ideas.

I was attacking this problem from the perspective of automatically testing an application that uses Dear ImGui, not testing Dear ImGui itself. I proposed a separation of concerns between documentation/scaffolding, automation, and testing because if they're separated this way, the functionality could be reused not only for documenting and testing ImGui itself (treating the demo app as an application to be tested), but also applications that leverage ImGui. Otherwise, desktop application automation usually relies on accessibility APIs, which are platform specific (or in the case of Qt, highly problematic).

I'll happily bow out, but I'd be curious if anyone else thought of it this way.

[ocornut]

@gerryhernandez You are absolutely right and this was omitted from discussions. The scaffolding developed for testing Dear ImGui should be reusable for testing Dear ImGui applications once the system is mature enough. This is an obvious target to aim for and will be kept in mind.

[ocornut]

Worked on a early draft of an automation system today.

API looks like e.g.

ctx->ItemOpen("ImGui Demo" "Widgets");
ctx->ItemOpen("ImGui Demo" "Basic");
ctx->ItemClick("ImGui Demo" "Basic" "Button");
[...]

The calls are synchronous blocking calls which are super easy to write.

void    ImGuiTestContext::ItemClick(const char* str_id)
{
    MouseMove(str_id);
    MouseClick(0);
}

void    ImGuiTestContext::ItemOpen(const char* str_id)
{
    const ImGuiTestLocateResult* result = ItemLocate(str_id);
    if (result->Window->DC.StateStorage->GetInt(result->ID, 0) == 0)
        ItemClick(str_id);
}

etc.

There's a lots more work to do but it's nice to have a minimum base.

[franciscod]

Wow, that looks really easy to use! Maybe the MouseMove function could be tweaked so the movement is a bit less robotic than linear but works great!

[ocornut]

A little update on automation. Those are rough notes mostly to convey that this idea is very well alive, and I'm not really going into details as to how this is working. I'd like to move this to a new topic when this is mature enough to be a thing, for now I am happy to keep it buried within this old topic :)

I have a little framework which can be used to register and run tests. This is running in a separate imgui-test repository (not yet public). Whenever possible, the framework tries to simulate user inputs.

This is all meant to be used to test dear imgui itself and to test applications using dear imgui.

Command-line version

Things can run completely blind, on the command-line, or in the interactive applications. Automation can run at maximum speed (teleport mouse, viewer runs at max framerate) or at human watchable speed.

The fastest is the blind command-line version:

test.exe -nogui -nopause all

---

Code coverage.

I would like to use the system to get some decent code coverage of imgui. With the current system it'll soon become easy to at least exercise a good portion of the core and demo code. While I don't expect code coverage to provide any great guarantee that things are working correctly, it seems like a baseline of sanity to at least have that. In fact I already found some regressions in recent imgui commits with what I have in place. For imgui+demo this would using a mixture of manually scripted interaction and discovery API (e.g. "get me all the items at this level of the ID stack, then click them"). Also looking at monkey-testing types of API ("click everywhere", "click randomly [in this window?]").

With high-level constructs we can currently do:

// Uncollapse everything in the demo window
t = REGISTER_TEST("demo", "demo_cov_auto_open");
t->TestFunc = [](ImGuiTestContext* ctx)
{
    ctx->SetRef("ImGui Demo");
    ctx->ItemOpenAll("");
};

// Open and close all the examples/help windows
t = REGISTER_TEST("demo", "demo_cov_apps");
t->TestFunc = [](ImGuiTestContext* ctx)
{
    ctx->SetRef("ImGui Demo");
    ctx->MenuClick("Menu/Open Recent/More..");

    ctx->MenuCheckAll("Examples");
    ctx->MenuUncheckAll("Examples");
    ctx->MenuCheckAll("Help");
    ctx->MenuUncheckAll("Help");
}

But it can also be more explicit:

        ctx->SetRef("ImGui Demo");
        ctx->ItemOpen("Help");
        ctx->ItemOpen("Configuration");
        ctx->ItemOpen("Window options");

            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Main menu bar");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Console");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Log");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Simple layout");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Property editor");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Long text display");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Auto-resizing window");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Constrained-resizing window");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Simple overlay");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Manipulating window titles");
            ctx->MenuAction(ImGuiTestAction_Check, "Examples/Custom rendering");

High-level functional testing

This would be using mostly user-supplied code, with explicit actions and checks/assertions. This is the sort of thing I have now:

    t = REGISTER_TEST("demo", "console");
    t->TestFunc = [](ImGuiTestContext* ctx)
    {
        ctx->SetRef("ImGui Demo");
        ctx->MenuCheck("Examples/Console");

        ctx->SetRef("Example: Console");
        ctx->ItemClick("Clear");
        ctx->ItemClick("Add Dummy Text");
        ctx->ItemClick("Add Dummy Error");
        ctx->WindowClose();

        IM_CHECK(false); // Will fail the test
    };

t = REGISTER_TEST("window", "auto resize collapse");
t->GuiFunc = [](ImGuiTestContext* ctx)
{
    ImGui::Begin("Test Window", NULL, ImGuiWindowFlags_NoSavedSettings | ImGuiWindowFlags_AlwaysAutoResize);
    ImGui::Text("Some text\nOver two lines\nOver three lines");
    ImGui::End();
};
t->TestFunc = [](ImGuiTestContext* ctx)
{
    ImGuiWindow* window = ImGui::FindWindowByName("Test Window");
    ctx->SetRef("Test Window");
    ctx->WindowSetCollapsed(false);
    ctx->LogVerbose("Size %f %f, SizeFull %f %f\n", window->Size.x, window->Size.y, window->SizeFull.x, window->SizeFull.y);
    IM_CHECK(window->Size == window->SizeFull);
    ImVec2 size_full_when_uncollapsed = window->SizeFull;
    ctx->WindowSetCollapsed(true);
    ctx->LogVerbose("Size %f %f, SizeFull %f %f\n", window->Size.x, window->Size.y, window->SizeFull.x, window->SizeFull.y);
    ImVec2 size_collapsed = window->Size;
    IM_CHECK(size_full_when_uncollapsed.y > size_collapsed.y);
    IM_CHECK(size_full_when_uncollapsed == window->SizeFull);
    ctx->WindowSetCollapsed(false);
    ctx->LogVerbose("Size %f %f, SizeFull %f %f\n", window->Size.x, window->Size.y, window->SizeFull.x, window->SizeFull.y);
    IM_CHECK(window->Size.y == size_full_when_uncollapsed.y && "Window should have restored to full size.");
    ctx->Yield();
    IM_CHECK(window->Size.y == size_full_when_uncollapsed.y);
};

I have several problems with that sort of test, the main one being that it can be really tedious and time consuming to create and maintain tests. I don't really have a silver bullet right now but I will keep massaging the design. However tedious this is, having core imgui being test would help further development a lot.

This will probably bring easier and faster benefits when testing end-user applications. Testing core imgui is a little harder, we need to be checking for a bunch of frame-to-frame race condition, edge cases, one-frame glitches, that are less important to test for end-user applications.

---

Capturing/screenshots

As mentioned in the old messages I would like to rely on this system to automate the creation of pictures and/or video. This could be useful for example to make it easy to regenerate documentation screenshots from a given piece of code, and keep them updated at all time. You could imagine a continuous integration system that took screenshots of all tools, and if an image changed someone would get notified that the documentation needs updating?

So the system needs to include high primitive, e.g. ScreenshotWindow("ImGui Demo"); that could be paused as part of the scripts.

---

Performance testing

The system should be usable to easily setup deterministic use cases (e.g. opening bunch of windows, with deterministic position/size/state), disabling hooks to reduce overhead, do basic measurement which could be used to track changes over time. Probably output performance data in a standard format. As with functional testing, the idea would be to build an array of performance tests.

---

That's a general overview. I still have a bunch of technical issues to solve to make this useful, and I predict we won't be able to stabilize the API for a while so mid-term this will primarily be useful for helping me to test core dear imgui and the reduces the amount of bugs creeping in.

[ddovod]

Hi there. I've just found and read this thread. Sorry if I missed something, but here's another perspective of how it could be implemented (some sort of such a system was used on my primary work). The main idea is to create replays of inputs, track imgui activity using logs/image or video capture, and then replay them. I'm not sure if it fits your needs, but probably it can be used as inspirations for smth else. So you're creating new test case, like Press The Button:

• You open the test app, it starts recording your activity inside it
• You open demo imgui window, press the button, some log message appears in the log
• You close the test app, it dumps all the data into test_case.bin file.

After that you open this app in "testing" mode, and now it reads all the test cases, reproduces all your inputs, checks log messages/screenshots to be the same as they was during recording. Smth like that. The main advantage - you can create test cases without writing code. As a reference you can check how it works in Visual Studio

[ocornut]

Hello @ddovod,

You are right I have omitted Capture/Replay playback from my post above because I haven't looked at this in detail yet but this is planned (may dabble with it soon, I've been actually working on the testing framework all day today).

Recording actually inputs (position etc.) is not good enough as they will break all the time (changing font size or some would break all tests), so we need to be recording higher-level of information (we don't want "Clicked at X;Y" but "Clicked on Window 1/OK". Also turning the first into the later requires some work and twisting of the internals).

Similar, visual comparaison are particularly brittle: e.g. it become not very feasible to test any large real world applications because they evolve all the time. Anything in the "ImGui Demo" window would be hard to test. Perhaps comparing at ImDrawData level can be preferable to allow finer filtering of what is being tested, avoid issue with undeterministic graphics rendering, and simplify the portability of tests, etc.

All of those aspects themselves are bound to require some experiment to provide the right solution with the right amount of flexibility, but I will definitively aim to have a record/playback system in there.

-Omar

[ocornut]

Small update showcasing a more specific case (this a script aimed at generating the screenshot used on the github main page). I'm running the script in "slow" mode here so the interaction are visible, but it can also run in a fraction of second.

[ocornut]

Still working on this in the background.

It is surprisingly difficult to get robust results given varying starting state. Normally most tests should run on a predictable starting state so this shouldn't be an issue, but making the high-level functions extra robust will help the system to be of use for user application, maybe even end-user exposed macros. (Still injecting mostly actual mouse/keyboard inputs.)

[elect86]

Hi Omar,

is this code already available somewhere?

[ocornut]

is this code already available somewhere?

It isn't. I can give you access to a repo but for now it is experimental and the API will keep changing for several months.

[elect86]

Yes, but only if I may start porting it on the jvm, otherwise I'd just wait

[ocornut]

It's way too experimental/early to be ported.

[ocornut]

Still working on this in the background (still very useful during development, stil too early to publish). Thanks to @rokups everything is now nicely wired to continuous integration systems via GitHub Actions (beta feature open to everyone), so we now build several examples on many platforms + the testing suite also runs on its own repository.

The core imgui side of things should be visible here: https://github.com/ocornut/imgui/actions

[gavlig]

Hello, may I request access to the said repo with testing framework as well? I understand that its api is unstable but it would still be better than hacking internal application state for tests.

[ocornut]

@gavlig Please e-mail me about this.

[adamierymenko]

I found this thread trying to figure out of this project supports screen readers for the blind, which requires some kind of accessibility API. I can't figure out the answer to this but I assume this is related. Anyone know? Should go in the FAQ. If not is this planned?

[ocornut]

@adamierymenko On my side this is not planned at the moment. There's a little bit of overlap with the work done for this issue, but overally the bulk of the work would be to figure out and interact with the dedicated OS accessibility api which are particularly complex to deal with. Someone more experienced with accessibility need would be more a adequate fit to look into this.

[elect86]

The CI integration is the cherry on top of the cake..

May I have access, Omar? I'd like to start playing around and see where it goes..

If this goes well, it will greatly help me to sync with your changes with much less pressure on my shoulders about introducing bugs/regressions, and also to find out which is the commit to blame when a test is failing (without me checking out again and again randomly commits to find it)

I read here it can even run headless..! This is really nice..

[ocornut]

A GIF of automation doing regression testings on some subtle tree node behaviors (testing combinations of: open on arrow flags, double click flags, with/without multi-select api, treenode as drag source, holding drag payload to open node, exacting reaction frame (some need to react on MouseDown other on MouseUp etc.):

[MouriNaruto]

I wonder to suggest imgui that add the support of MSAA or UIA for Windows platform because it can help lots of people.

[elect86]

Hey @ocornut,

may I ask you also access for one collaborators of ours (@Sylvyrfysh)?

[pthom]

Just a quick suggestion, which I hope is not too strange. imgui.h could be used as a source in order to generate external doc.

My idea stems from markdown preview enhanced, which enables to add diagrams, equations, table of content, etc to markdown.

Anyhow, I have made some test in another project, and I can obtain doc that looks like this: api.md

With a doc "source" which looks like this: api.src.md

A where a documented .cpp or .h file might look like this: source file with embedded markdown

And the doc generation script could be inspired from this: script