Provide aggregated tests for quarkus-platform

lburgazzoli commented 4 years ago

The quarkus-platform now includes the same tests as we have here but it does not seems very sustainable in term of maintenance and resources required to validate the quarkus-platform.

We should create some aggregated test that validates multiple components in a single test

ppalaga commented 4 years ago

I was also thinking of how the platform can maintain our tests in the long term. My idea was to produce some metadata on our side (basically a list of our itest project GAVs) that the platform could leverage to generate/validate the modules in their source tree. I was thinking of adding a mojo to rpkgtests-maven-plugin to implement that.

The rpkgtests:rpkgtests mojo could also use the metadata so that the testJars do not need to get configured manually.

This would be easier for us (we'd keep working as before, we'd just add a maven module that would generate the metadata). Not sure this would be acceptable for the platform.

ppalaga commented 4 years ago

My idea was to produce some metadata on our side (basically a list of our itest project GAVs) that the platform could leverage to generate/validate the modules in their source tree.

This is implemented now. Updating the tests in the platform is not that a big issue with the new tooling. I also would not say the number of the tests is an issue for them. Their CI runs only in JVM mode.

Having said that, I do not see an immediate need to improve anything in this area.

ppalaga commented 4 years ago

Now after a couple of Camel Quarkus upgrades in the Platform and after @galderz has detected quite a few issues through our tests there, I'd like to update my standpoint as follows:

Upgrades and maintenance of our tests in the platform is not an issue with the existing tooling. mvn validate -Pregen-camel -N regenerates the tests from scratch, so that new tests come and stale tests disappear. It is fast and reliable. Ad hoc fixes can be (or indeed are) done in the pom.xml template: https://github.com/quarkusio/quarkus-platform/blob/master/src/rpkg-templates/run-tests-module-pom.xml#L31-L41
Having all/nearly all our tests in the platform is a good thing. They help to uncover real issues, typically the ones caused by different dependency versions on the platform. Different dependency versions on the platform is a reality that will hardly change. There are two main sources of it:

(1) quarkus-bom is imported before our BOM and (2) other 3rd party BOMs, such as Kogito BOM are imported before our BOM on the platform.

Thus dependencies managed in all those 3rd party BOMs get a higher precedence that the ones managed by us.

Given the above I am against bothering with creating any reduced set of tests for the platform and I vote for closing this issue.

jamesnetherton commented 4 years ago

Having all/nearly all our tests in the platform is a good thing

Agreed, but what happens if we do eventually support another 100+ extensions. We potentially have a negative impact on the platform side by adding an ever increasing amount of stuff.

ppalaga commented 4 years ago

what happens if we do eventually support another 100+ extensions. We potentially have a negative impact on the platform side by adding an ever increasing amount of stuff.

I'd leave it to the platform to define their limits and complain when we exceed them.

lburgazzoli commented 3 years ago

@ppalaga @jamesnetherton I'd close this issue and think about this issue if/when there will be complains from quarkus, what do you think ?

jamesnetherton commented 3 years ago

@ppalaga @jamesnetherton I'd close this issue and think about this issue if/when there will be complains from quarkus, what do you think ?

Yep - I agree. Let's close.

apache / camel-quarkus

Provide aggregated tests for quarkus-platform #413