Test DGPv2 example projects

[adam-enko]

Test the DGPv2 examples by comparing using golden testing.

The example projects are executed using Gradle TestKit. The generated Dokka output is compared against data committed into the directory ./dokka-integration-tests/gradle/src/testExampleProjects/expectedData/

To re-emphasise, everything in testExampleProjects/ is test data, and these are the remaining modified files:

KT-71346

Dependencies

• This PR requires https://github.com/Kotlin/dokka/pull/3820

[abdulowork]

It seems the GH UI is lagging pretty heavily for me due to the test fixture; rebasing the fixture to the first (or last) commit and making a stacked branch will probably reduce the lag

[abdulowork]

I like that this PR expands on E2E testing in Dokka; however, is it really feasible to test the generated html output in the long term? Imho, at the E2E level, html is more of an implementation detail. I imagine if someone is going to implement a major Dokka redesign in the future, they will have to regenerate these html files after every UI change.

[adam-enko]

I like that this PR expands on E2E testing in Dokka; however, is it really feasible to test the generated html output in the long term? Imho, at the E2E level, html is more of an implementation detail. I imagine if someone is going to implement a major Dokka redesign in the future, they will have to regenerate these html files after every UI change.

For some context, Dokka's output formats don't change very often. I've used a similar sort of direct file comparison testing in Dokkatoo for over a year, and I haven't had any instability.

We can work on making the reference files easier to update, but that's something I'd rather look at if/when it becomes a problem (https://xkcd.com/1205/).

[whyoleg]

For some context, Dokka's output formats don't change very often. I've used a similar sort of direct file comparison testing in Dokkatoo for over a year, and I haven't had any instability.

In coming days and months we will have HTML changes: related to nav bar, ToC, search, may be more.

Let's do an experiment (mind or real). We have a PR: https://github.com/Kotlin/dokka/pull/3734 What are the ALL steps needed so that the developer, who will want to merge this PR need to do?

[abdulowork]

I still want to challenge the idea that we should test html output:

Dokka's output formats don't change very often

but then is there a point in testing the html output that doesn't really change? In general I would expect E2E testing to be rather carried out using a UI testing framework like Selenium

[adam-enko]

I don't think, that we should merge this PR without some way to update expectedData. E.g #3821 (comment)

There is already a way to update the expected data

https://github.com/Kotlin/dokka/pull/3821#discussion_r1780789277

[adam-enko]

I still want to challenge the idea that we should test html output:

Dokka's output formats don't change very often

but then is there a point in testing the html output that doesn't really change? In general I would expect E2E testing to be rather carried out using a UI testing framework like Selenium

Yes, there is a point. We want to make sure that changes in DGP/AGP/Gradle/KGP don't affect the HTML output.

[adam-enko]

What are the ALL steps needed so that the developer, who will want to merge this PR need to do?

1. Run tests.
2. See the error message.
3. Copy the files from the test directory into the expected-data directory.

It takes a few minutes to do.