REST stages occasionally fail with `Address already in use` exception

camunda / camunda-bpm-platform

Flexible framework for workflow and decision automation with BPMN and DMN. Integration with Quarkus, Spring, Spring Boot, CDI.

https://camunda.com/

Apache License 2.0

4.11k stars 1.55k forks source link

Closed tmetzke closed 1 year ago

tmetzke commented 1 year ago

All REST stages (i.e. all parallel stages in CE Main Pipeline in stage Engine Rest UNIT tests) are stable regarding Address already in use exceptions.

Occurred in CE Main Pipeline occasionally in 7.20 first
No stack trace in the stages, failing tests only display java.net.BindException: Address already in use

### Pull Requests
- [ ] https://github.com/camunda/camunda-bpm-platform/pull/3536

[ ] Does this ticket need a QA test and the testing goals are not clear from the description? Add a Dev2QA handover comment

tmetzke commented 1 year ago

The REST unit tests spin up a local test server using the respective environment (i.e. Netty, Glassfish Grizzly, Tomcat, Undertow).
The underlying OS can be too slow to release the port/address that was blocked by a test server instance after the server has run through its stop method. In effect, the port release is asynchronous but cannot be waited upon effectively via any API.
Thus, it can occasionally happen that the test server tries to start although the port is not released by the OS yet.
This is the behavior we observed in the web app unit tests as well.
The servers we use fail to start when the address is still in use. There is no retry mechanism they bring along to counter this effect.

For consistency, I decided to go with "wait and retry" here - the same as in the web app unit tests.

tasso94 commented 1 year ago

@tmetzke moved comment to backport ticket