search

getting-things-gnome / gtg

Getting Things GNOME! trunk
https://wiki.gnome.org/Apps/GTG
GNU General Public License v3.0
552 stars 166 forks source link

Better testing, with automated UI integration tests [RFC] #399

Open diegogangl opened 4 years ago

diegogangl commented 4 years ago

This is an issue to discuss how we could improve our automated testing (or lack of). Some things that have been suggested:

nekohayo commented 4 years ago

Having greater test coverage, especially UI "integration testing" (i.e. what a user would actually encounter in practice) with Dogtail (I don't know if there's anything better these days on the Linux desktop, but I'm guessing this is still the go-to thing to get it done easily) would allow refactoring and merging improvements with the proven assurance that things are improving (or at least not regressing).

I think covering the Task Editor should be the first target of the test suite, because that's where a lot of stuff can go wrong in different ways, and it's difficult to manually test everything all the time. Dogtail would be particularly useful if it can handle rich text formatting somehow, for the gazillion potential little bugs and gotchas in GTG's "Task Editor".

The main window ("Task Browser") is less urgent, in part because it is supposed to already be covered by liblarch's test suite (?) and in part because Diego is considering providing a new GtkListBox-based widget backend for liblarch instead of something based on GtkTreeView.

I'm considering this ticket a low-hanging-fruit because in theory Dogtail is so easy and fun to use (you tell it that the GUI should react this way or that way, or have this widget or this text or this property after typing some text or clicking some button, and if it doesn't match your expectations, you raise a test failure)... that any new contributor could work on this (and provide a very valuable contribution by doing so).

If someone is interested in giving Dogtail a try for GTG, here are some examples of how it was done in Pitivi many years ago, with an old version of Dogtail (in the "common.py" file there I had put often-used UI functions that could be reused across tests, and also various tricks to significantly speed up Dogtail by saving widget references into variables, therefore not needing to traverse the whole widget tree everytime you want to access something). There are also some more recent examples from the Dogtail project itself.

Note that Dogtail is easier to use on Xorg/X11 than Wayland (see this ticket), which makes it sound like it is not impossible to use it on Wayland either, due to the existence of the Ponytail bridge daemon.

nekohayo commented 6 months ago

Fast-forward nearly 4 years later:

It seems to me that if Dogtail comes back to life, it will probably be years in the making. Is this the only possibility, or is there something else we were not aware of?

nekohayo commented 4 months ago

I got confirmation that Ponytail on top of Dogtail remains the official way forward, both from the existence of this recent Fedora Magazine tutorial article and a GNOME devs discussion today. Quoting @modehnal from that discussion (who is the author of that article, and probably the best person we can ask about the state of things if we run into trouble trying to get this working for GTG):

We still actively maintain dogtail (we just dont have it in pypi in the latest version although I would like that to be there too) and make sure it is able to be used for gtk4. If you would like to test, make sure to follow the setup of dogtail from devel/wayland branch and gnome-ponytail-daemon. From that point you should have no issues with dogtail and gtk4. But beware that it is not yet in the best state everywhere. For example take a look at this issue I discovered recently...

The future of dogtail (that I imagine) is described in the lengthy full solution. In the future I want to get rid of dogtail in favor of Atspi only wrapper. Which is both as simple as possible while using only Atspi and also user friendly like dogtail is.

I already have a prototype (lightweight clone of dogtail) that I made so that I can debug Atspi quicker and provide reproducers for upstream so they can look at the issues and have no blockers from our testing environment

Emmanuele mentioned:

One thing in GTK4 that impacts dogtail and UI testing through the accessibility interfaces is that we (well: I) have been pretty adamant in not implementing any "set this value programmatically" bit from AT-SPI; we only have actions, though the AtspiAction interface is really limited. The other thing being: coordinates systems in AT-SPI are a disaster zone

Michal replied:

Perhaps, but as far as rawhide goes, I was pleasantly surprised how well everything works from AT-SPI side. Aside from a few issues I filed everything works very well both on rawhide and rhel-10. Lukas does amazing work for Accessibility side of things and I noticed that right away, many objects were correctly labeled, so far I was able to remove 500+ lines of workarounds from GNOME Control Center suite

I also noticed the new accessibility architecture possibility you mention in your issue. I have sent an email to the author where I expressed my interest in that. If something comes from that I will for sure write a wrapper around it for dogtail (or most likely in the new project)

modehnal commented 4 months ago

@nekohayo Hello!

I was curious about this project and looked into it.

It seems quite nice and I was surprised how accessible it is.

So I took a few minutes and Fedora rawhide VM I had on hand and I put together a starting point for your automation.

The setup I did was just to install GTG from flatpak and let our automation stack to take care of the rest.

I put the code here https://github.com/modehnal/gtg_automation_suite , make sure to look at the review folder where I put a full html report page and a video of your project automation. Fully functional with dogtail and ponytail.

Please beware that unless you are familiar with our automation stack, do not run it on your main machines, we only use VMs for our testing. For more information look at the full solution https://modehnal.github.io/

What you will find in that repository works nicely (as you can see in the video), contains fully set up solution for GTG with our automation stack. Hopefully you will find it useful or at least some parts of it! :smile:

Good luck!

nekohayo commented 3 months ago

Oh wow, nice! Thanks for the example. Hopefully this could run nicely with GTG's launch.sh script with the -S (dataset) parameter to use one of our sample datasets, which would guarantee that you absolutely cannot wreck user data even if running this on your main machine, which I guess would make it much more compelling to integrate in to the main repo / as part of the main test suite.

One question on my mind is, we've seen changes happening very recently with the accessibility stack with Matt Campbell's work, with not only AccessKit but also the new "Newton" project. The most recent situation summary I know of is https://lwn.net/Articles/971541/

Does AccessKit and Newton change anything for:

modehnal commented 3 months ago

I am not yet familiar with Newton project so I will read about that and get up to speed about the Matt's progress.

As of now for the:

Does AccessKit and Newton change anything for: Dogtail/Ponytail (i.e. will you folks have to undergo a period of rewriting various things etc.)?

Indeed, if something get released and will get to Fedora I will explore that and things will need to get rewritten for sure.

The dogtail uses with pyatspi2 and Atspi python modules, so dogtail would need to be adapted for the new API, but I do not think we want that to happen. I imagine we will want to have 2 projects at the end, one that uses Atspi and one that uses the new accessibility stack. If nothing else it should provide a choice.

If there is going to be a decision that only of those will be supported from certain point in time than yeah, we will do a rewrite to the supported API and maintain only one project.

GTG, i.e. should we be waiting for the dust to settle on all of the above, or something that is written today with the current implementation would still work exactly the same a year or two from now?

If year from now you will still be on systems with GNOME Shell (which provides introspection for gnome-ponytail-daemon) and Atspi/pyatspi2 (which dogtail requires) than it should work without any issues. We have a lot of automation suites built on this for most of the GNOME Desktop environment.

Lets say the new API will get released tomorrow, Atspi and pyatspi2 will still exist and will work for some time, its not trivial to remove entire parts of the system from any distribution. Note that Orca uses Atspi just like dogtail does, just from the other end. (They use mouse events - provide coordinates - and get "strings". Dogtail uses queries that give "strings" and gets coordinates on which we generate mouse events) So I do not think anyone can justify to remove Atspi as of now.

If time allows it, give it a try but do not bet everything on it.

I do not like dogtail. It does things I/we need so I tolerate it for now.

I wanted to do a big rewrite because the code base is not in a good state but I could not justify the time it would take and there are also fears that all the work I would put into it would be for nothing if it gets removed/replaced.

I hope that the new accessibility stack will be a win and accessibility itself will be more stable and we can build upon it.

modehnal commented 3 months ago

So I read summary Article about Matt's progress, I went through his branches in various projects. I believe I will be able to make use of what is there to test basic functions.

I also talked about dogtail with the main maintainer, vhumpa. It seems I misunderstood how "fixed" dogtail source code is. He is not against doing major changes to fix some issues and make improvements. He is planning for a long time a 1.0 release, commiting to wayland only, but was not able to get to it because of other responsibilities. We brainstormed a little and agreed on some points and will rewrite dogtail properly once we have time.

The current way forward is this once time allows it.