Verify usage of gobject regression tests

[badcel]

There is already some c code to test behavior of gobject code:

• https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/main/tests/scanner/meson.build#L465
• https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/main/tests/scanner/regress.c
• https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/main/tests/gimarshallingtests.c
• https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/main/tests/meson.build#L141

This probably requires to build gobject as part of CI?

does gir.core should depend on external test code?

[ylatuya]

Hi! Over the last year I have been working on fixing the GtkSharp's bindings generation to be used by GStreamer and I have been using GObject Introspection's Regress library to write both integration and unit tests. It has helped to fix many issues in corner cases that this library exposes. Working with GtkSharp is a real pain and I think GitCore is the way to go and I would like to contribute all these changes to GitCore.

The test infrastructure has the following component:

• Regress library: the library and the introspection data are compiled with meson creating the shared library the .typelib and the .gir that is consumed by the tests.
• Integration tests: the integration tests generate the bindings from Regress-1.0.gir and compile the resulting library using Microsoft.CodeAnalysis.CSharp.CSharpCompilation verifying compiler errors and warnings.

• Unit tests: the regress library also comes with unit tests, these unit tests are run through the generated bindings validating that the generated code not only can be compiled but also marshals the input parameters and the return values correctly.

  Let me know if this is something interesting for the project and I will be happy to help with that.

[badcel]

I'm not sure if I want to depend on external test code. This is why this issue is called "Verify...".

To discuss this you are welcome to join the matrix channel which is referenced in the readme.

I don't know if you are aware but there are already several tests which execute C code:

• C library
• Unit Tests

and the projects one level above this one.

Currently I would tend to say that I don't want to depend on external test code. Otherwise contributing tests would involve working on several projects and be dependent on the will to add tests which are perhaps not wanted. Additionally it is harder to check them again as it is a different project. Additionally what happens if tests are added / removed?

[ylatuya]

The Regress library covers many scenarios and corner cases that might not be covered by the test suite around GirTestLib as it's being used for regression tests in GObject Introspection itself. This is the code that GtkSharp generates from that library: https://github.com/GLibSharp/GtkSharp/tree/master/Source/tests/generator/regress/generated/Regress

It does not need to depend on external code, the source code of the library can be imported in the repo and built in the same way GirTestLib is built. The code for the Regress library will not change and would only require to be synced upstream from time to time, although

The repository layout would look like:

• src/Native/RegressLib:
  • libregress: 'regress.c', 'regress.h', 'annotation.c', 'annotation.h', 'foo.c', 'foo.h', 'drawable.c', 'drawable.h'
  • libutility: 'utility.c', 'utility.h'
  • meson.build
• src/Tests/Libs/Regress-0.1.Tests:
  • Regressions tests: generate the bindings and validate the number of objects, methods, callbacks, enums, etc... that are genereted.
  • Unit tests: runs the test suite of the library.

All of the Regress library cases can be re-implemented in GirTestLib, but short-term it can provide a quick win by adding coverage and helping to implement missing features and corner cases that might be handled correctly yet.

[badcel]

What do you mean by this:

Regressions tests: generate the bindings and validate the number of objects, methods, callbacks, enums, etc... that are genereted.

The current GirTest C# lib is build as a library during build like every other supported library (e.g. GTK). It sounds like you somehow want to generate bindings in a different way for the regression tests?

I don’t fully get how you imagine your proposal. Are the c# tests written by hand?

The current approach is to add tests for all newly supported scenarios. Already existing scenarios get covered if there is a bug reported or as a drive by test of a new feature. This will slowly increase the test coverage and not bring development to a halt.

I think I still don’t want to have foreign code to depend on in this repository. I read from your comment that you kind of agree with this.

If you are interested in finding problems in the currently generated code via the regression tests those findings are very welcome. I think every bug caught should result in a new unit test through GirTest lib.

I don’t know how much work it is for you to get something going. But you could for example include gir.core on your on repository as a submodule and have the regression tests running as part of your repository and report detected problems here. And we fix them one by one and add tests along the way.

What do you think?

[ylatuya]

I have been busy with other projects and, I didn't have time to reply.

The Regress library provides a comprehensive set of all the features supported by GObject Introspection. GirTest serves a similar purpose but it's still incomplete: it's hard to know what's missing to be supported and all the missing bits need to be re-implemented (eg: fundamental types). Including the Regress as a library, in a similar way GirTest or Gtk, can provide some quick benefits:

• The code generator is tested against a .gir that contains almost all the possible features and variations until GirTest is not feature complete.
• You can have tests for use cases that are not supported in GirTests without having to do an implementation from scratch, such as tests for fundamental types

The Regress library also has a set of unit tests that validates different aspects of the implementation, for different types input/output parameters, signals, callbacks and so on. In the same way there is a test suite for Gtk or GirTest, there would be one for Regress.

The inclusion of Regress library will look like:

• Native library in Native/Regress
• Library in Libs/Regress-1.0
• Test in Tests/Libs/Regress-1.0.tests

[badcel]

I get the layout you are proposing and it is in line with the rest of the system.

But I'm not even sure about the license of the regress code as we are currently MIT.

Additionally I won't be able to review or maintain a 30.000 lines changed pull request.

Please see my final question in my last comment:

If you are interested in finding problems in the currently generated code via the regression tests those findings are very welcome. I think every bug caught should result in a new unit test through GirTest lib.

I don’t know how much work it is for you to get something going. But you could for example include gir.core on your on repository as a submodule and have the regression tests running as part of your repository and report detected problems here. And we fix them one by one and add tests along the way.

What do you think?

[badcel]

For example you could take a topic you like (e.g. GArray) and add some newly written tests and make it actually useable.

This would be a small topic which could be split into like 3 PRs (one per array type). I could see how your code looks like on a small scope and we can work it out together so it fits into the project.

[ylatuya]

I understand your point perfectly in regards to including the Regress library in the project.

For my initial evaluation, I would still like to test how gir.core performs with the Regress library at least to get a grasp of what's missing/failing in order to proceed later on with shorter and focused PR's.

This is the branch I am working on: https://github.com/ylatuya/gir.core/tree/regress without the intention to merge it and only for this initial evaluation goal.

[badcel]

Great to hear. Let's see how things go along and will develop over time.

You should be able to build the library similar to the GirTest library: GenerateGirTestLib

Afterwards it should be straightforward to generate the regress bindings if you just add the name of the Gir file as an argument to the GenerateLibs script. See the build documentation.