feat: Sync timeouts for applications (#6055)

[andrii-korotkov-verkada]

Helps with #6055

Introduces a controller-level configuration for terminating sync after timeout.

Checklist:

• [x] Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• [x] The title of the PR states what changed and the related issues number (used for the release note).
• [x] The title of the PR conforms to the Toolchain Guide
• [ ] I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• [ ] I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• [ ] Does this PR require documentation updates?
• [ ] I've updated documentation as required by this PR.
• [x] I have signed off all my commits as required by DCO
• [x] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• [x] My build is green (troubleshooting builds).
• [ ] My new feature complies with the feature status guidelines.
• [x] I have added a brief description of why this PR is necessary and/or what this PR solves.
• [x] Optional. My organization is added to USERS.md.
• [ ] Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell[bot]]

🔴 Preview Environment stopped on Bunnyshell

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🔵 /bns:start to start the environment
• :rocket: /bns:deploy to redeploy the environment
• :x: /bns:delete to remove the environment

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@6d8d32f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
controller/appcontroller.go	60.00%	3 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #20816 +/- ## ========================================= Coverage ? 55.16% ========================================= Files ? 324 Lines ? 55268 Branches ? 0 ========================================= Hits ? 30486 Misses ? 22165 Partials ? 2617 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[rumstead]

Nice! Do you think we can add a few e2e tests?

[andrii-korotkov-verkada]

I have TestSyncTimeout, but can also add end-to-end if necessary. Though I'm not sure they'd add more value, since they'd be replicating the logic from TestSyncTimeout. What do you think?

[rumstead]

I have TestSyncTimeout, but can also add end-to-end if necessary. Though I'm not sure they'd add more value, since they'd be replicating the logic from TestSyncTimeout. What do you think?

I was thinking that deploying an application that will never leave progressing and being able to make sure it the sync status fails would be a good one to have. If you think the unit tests are solid, rock on!