search

paketo-buildpacks / apache-tomee

A Cloud Native Buildpack that contributes Apache Tomee and Process Types for the application server
Apache License 2.0
0 stars 3 forks source link

Integration tests are flaky and frequently fail #237

Open dmikusa opened 10 months ago

dmikusa commented 10 months ago

Expected Behavior

  1. Integration tests should consistently pass
  2. We should not have integration tests. It's difficult to mock out other buildpacks and external factors for a reliable integration test. If we can't make it reliable, then we should just remove it.

Current Behavior

See https://github.com/paketo-buildpacks/apache-tomee/actions/runs/7378265147/job/20073179955#step:9:99 as an example.

Possible Solution

OR

Steps to Reproduce

Pick one of the integration tests & re-run it. It seems to fail around 50% of the time.

Motivations

It causes hassles when evaluating PRs and wastes time. No one needs that.

loewenstein commented 10 months ago

It is fair to evaluate individual test cases for their weight-to-worth ratio. However, I don't think that removing testing entirely - and removing a whole test category like integration tests comes really close to that I believe - is a good idea.

@c0d1ngm0nk3y @pbusko @modulo11 @phil9909 we do run integration tests on component and composite buildpack level and don't experience a failure rate like @dmikusa mentions, right? Do we have anything to share to help improve the test success rate for Paketo?