feat: Add retry on success capability for Store and Forward

[lenny-goodell]

closes #1526

If your build fails due to your commit message not passing the build checks, please review the guidelines here: https://github.com/edgexfoundry/app-functions-sdk-go/blob/main/.github/CONTRIBUTING.md

PR Checklist

Please check if your PR fulfills the following requirements:

• [x] I am not introducing a breaking change (if you are, flag in conventional commit message with BREAKING CHANGE: describing the break)
• [x] I am not introducing a new dependency (add notes below if you are)
• [x] I have added unit tests for the new feature or bug fix (if not, why?)
• [x] I have fully tested (add details below) this the new feature or bug fix (if not, why?)
• [x] I have opened a PR for the related docs change (if not, why?) https://github.com/edgexfoundry/edgex-docs/pull/1328

Testing Instructions

Run non-secure Edgex stack with Device Rest and MQTT Broker from compose builder

make run no-secty ds-rest mqtt-broker

Build ASC with this branch of APP SDK Modify the ASC mqtt-export profile as follows:

• Enable Store and Forward

    StoreAndForward:
      Enabled: true

• Set MaxReconnectInterval for MQTT Export

  MaxReconnectInterval: "5s"

• Enable PersistOnError for MQTT Export

  PersistOnError: "true"

  Run ASC with DEBUG logging

  WRITABLE_LOGLEVEL=DEBUG ./app-service-configurable -p=mqtt-export -cp -d -o 

  Verify Store and Forward started by checking for this log message.

  msg="Starting StoreAndForward Retry Loop with 5m0s RetryInterval and 10 max retries. 0 stored items waiting for retry."

  Send "1234" to localhost:59986/api/v3/resource/sample-numeric/int using postman or curl Verify data was exported successfully via this log message

  "Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"

  Verify Retry was not triggered since no failed data stored for retry. I.e. this message NOT in logs

  msg="Triggering Store and Forward retry of failed data "

  Stop the MQTT Broker Send "1234" to localhost:59986/api/v3/resource/sample-numeric/int using postman or curl 5 times Verify each was saved for later retry

  msg="MessageBus Trigger: Failed to process message on pipeline(s): 1 error occurred:\n\t* in pipeline 'default-pipeline', connection to mqtt server for export not open, persisting Event for later retry\n\n"

  Stop ASC Restart MQTT Broker Restart ASC Verify Store and Forward started and has 5 items stored for retry.

  msg="Starting StoreAndForward Retry Loop with 5m0s RetryInterval and 10 max retries. 5 stored items waiting for retry."

  Send "1234" to localhost:59986/api/v3/resource/sample-numeric/int using postman or curl Verify data was exported successfully via this log message

  "Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"

  Verify the successful send triggered the retry of failed items.

  msg="Triggering Store and Forward retry of failed data "
  msg="5 stored data items found for retrying"    
  msg="Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"
  msg="Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"
  msg="Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"
  msg="Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"
  msg="Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"
  msg=" 5 stored data items will be removed post retry"
  msg=" 0 stored data items will be updated post retry"

  Send "1234" to localhost:59986/api/v3/resource/sample-numeric/int using postman or curl on final time Verify data was exported successfully via this log message

  "Sent 377 bytes of data to MQTT Broker in pipeline 'default-pipeline' to topic 'edgex-export'"

  Verify Retry was not triggered since no failed data stored for retry. I.e. this message NOT in logs

  msg="Triggering Store and Forward retry of failed data "

New Dependency Instructions (If applicable)

[codecov-commenter]

Codecov Report

Attention: 39 lines in your changes are missing coverage. Please review.

Comparison is base (0f93857) 66.63% compared to head (131e15b) 66.27%.

Files	Patch %	Lines
internal/runtime/storeforward.go	47.16%	26 Missing and 2 partials :warning:
internal/appfunction/context.go	0.00%	6 Missing :warning:
internal/runtime/runtime.go	50.00%	2 Missing and 1 partial :warning:
pkg/transforms/mqttsecret.go	0.00%	2 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1536 +/- ## ========================================== - Coverage 66.63% 66.27% -0.36% ========================================== Files 36 36 Lines 3198 3238 +40 ========================================== + Hits 2131 2146 +15 - Misses 929 951 +22 - Partials 138 141 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[lenny-goodell]

@judehung, @cloudxxx8 , could you please review this PR and test it with your performance test to verify that it doesn't impact performance when data is being successfully exported. ie. the attempt to trigger retry after each successful send is minimal when there isn't any data stored to retry. Thanks!