add support for retrying pending changes

[wei-lee]

What

This PR introduces a new feature to allow the pending changes to be retried. The number of retries and the retry interval can be configured by developers.

Why

This is related to issue: https://issues.jboss.org/browse/RHMAP-21014.

What happens is that sometimes the custom sync handlers could fail occasionally. At the moment, when an error occurred, there is no retry mechanism and the pending change is simply treated as processed and later on the client data will be reverted. This will cause some confusion to both users and developers and the sync server itself should allow occasional failures from the sync handlers.

To test

• Run the new integration test, or
• Deploy a new cloud app and use this version of fh-sync. Then make sure the update custom handler will fail on the first call but should succeed on the subsequent calls. Run the sync app as normal, but you should see the data will be updated eventually in the backend and the client data is not reverted back.

ping @wtrocki @camilamacedo86 @shannon

[wtrocki]

@wei-lee World class work. I have tested that with integration tests (with modified parameters as well) and also in standalone server.

[wtrocki]

Good to go! Going to include this changes in the batch of sync changes that were done recently and retest that again.