Shutdown expunged resources cleanup executor properly, and allow other components to configure/start/stop on error

sureshanaparti commented 2 months ago

Description

This PR shutdowns expunged resources cleanup executor when obj is available (when config expunged.resources.purge.enabled is true), allows other components to configure/start/stop on error, and adds some logs in component lifecycle classes.

Noticed this exception with custom logs, the remaining components fails to stop after this exception.

WARN  [o.a.c.s.l.CloudStackExtendedLifeCycleStart] (SpringContextShutdownHook:null) (logid:) Error on stopping beans - null
java.lang.NullPointerException
        at org.apache.cloudstack.resource.ResourceCleanupServiceImpl.stop(ResourceCleanupServiceImpl.java:584)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$2.with(CloudStackExtendedLifeCycle.java:105)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:159)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.stopBeans(CloudStackExtendedLifeCycle.java:101)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.stop(CloudStackExtendedLifeCycleStart.java:32)
        at org.apache.cloudstack.spring.lifecycle.AbstractSmartLifeCycle.stop(AbstractSmartLifeCycle.java:49)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:234)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:54)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:373)
        at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:206)
        at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:129)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1069)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:993)

Fixes #9722

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI
[ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[x] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):

Mgmt2 service stopped =>

How Has This Been Tested?

Manually tested management server start & stop.

2024-09-23 18:20:51,111 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterServiceServletAdapter
2024-09-23 18:20:51,111 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterManagerImpl

[root@ol8 ~]# cat stopping-beans-check.txt | wc -l
665

stopping-beans-check.txt

How did you try to break this feature and the system with this change?

sureshanaparti commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 4.48%. Comparing base (b1f683d) to head (a6eb0b0). Report is 44 commits behind head on main.

:exclamation: There is a different number of reports uploaded between BASE (b1f683d) and HEAD (a6eb0b0). Click for more details.

HEAD has 1 upload less than BASE
| Flag | BASE (b1f683d) | HEAD (a6eb0b0) | |------|------|------| |unittests|1|0|

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #9723 +/- ## ============================================ - Coverage 15.77% 4.48% -11.30% ============================================ Files 5621 392 -5229 Lines 491564 32154 -459410 Branches 61174 5672 -55502 ============================================ - Hits 77562 1441 -76121 + Misses 405545 30707 -374838 + Partials 8457 6 -8451 ``` | [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/9723/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/9723/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.48% <ø> (+0.43%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/9723/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11181

sureshanaparti commented 2 months ago

@blueorangutan test

blueorangutan commented 2 months ago

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti commented 2 months ago

@blueorangutan package

sureshanaparti commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11182

DaanHoogland commented 2 months ago

@sureshanaparti , this looks like a good cleanup. I wonder what it fixes other than just the looks of the code, though. We have two shutdown issues:

prolonged time of shutdown
no status update for shutdown MSses does this address voth, @sureshanaparti ? (I can see you showed some evidence for the second)

sureshanaparti commented 2 months ago

@sureshanaparti , this looks like a good cleanup. I wonder what it fixes other than just the looks of the code, though. We have two shutdown issues:

prolonged time of shutdown

no status update for shutdown MSses does this address voth, @sureshanaparti ? (I can see you showed some evidence for the second)

@DaanHoogland this updates MS status to Down when service is stopped/shutdown. (it doesn't address prolonged time of shutdown)

blueorangutan commented 2 months ago

[SF] Trillian test result (tid-11539) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 62071 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9723-t11539-kvm-ol8.zip Smoke tests completed. 134 look OK, 2 have errors, 5 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_03_secured_to_nonsecured_vm_migration	`Error`	375.65	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	282.02	test_vm_life_cycle.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py

DaanHoogland commented 1 month ago

@blueorangutan test keepEnv