New Feature: Multi-arch Zones

nvazquez commented 2 months ago

Description

This PR introduces the multi-arch zones, allowing users to select the VM arch upon deployment.

Multi-arch zone support in CloudStack can allow admins to mix x86_64 & arm64 hosts within the same zone with the following changes proposed:

All hosts in a clusters need to be homogenous, wrt host CPU type (amd64 vs arm64) and hypevisor
Arch-aware templates & ISOs:
- Add support for a new arch field (default set of: amd64 and arm64), when unspecified defaults to amd64 and for existing templates & iso
- Allow admins to edit the arch type of the registered template & iso
Arch-aware clusters and host:
- Add new attribute field for cluster and hosts (kvm host agents can automatically report this, arch of the first host of the cluster is cluster's architecture), defaults to amd64 when not specified
- Allow admins to edit the arch of an existing cluster
VM deployment form (UI):
- In a multi-arch zone/env, the VM deployment form can allow some kind of template/iso filtration in the UI
- Users should be able to select arch: amd64 & arm64; but this is shown only in a multi-arch zone (env)
VM orchestration and lifecycle operations:
- Use of VM/template's arch to correctly decide where to provision the VM (on the correct strictly arch-matching host/clusters) & other lifecycle operations (such as migration from/to arch-matching hosts)

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI
[ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

nvazquez commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 15.05376% with 237 lines in your changes missing coverage. Please review.

Project coverage is 15.76%. Comparing base (679ce1a) to head (daec7bc). Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
api/src/main/java/com/cloud/cpu/CPU.java	40.00%	15 Missing :warning:
...n/java/com/cloud/resource/ResourceManagerImpl.java	0.00%	14 Missing :warning:
agent/src/main/java/com/cloud/agent/Agent.java	0.00%	13 Missing :warning:
...src/main/java/com/cloud/dc/dao/ClusterDaoImpl.java	40.00%	12 Missing :warning:
...java/com/cloud/agent/manager/AgentManagerImpl.java	0.00%	10 Missing :warning:
...n/java/com/cloud/template/TemplateAdapterBase.java	0.00%	9 Missing :warning:
...a/org/apache/cloudstack/util/CPUArchConverter.java	0.00%	7 Missing :warning:
.../cloud/storage/upload/params/UploadParamsBase.java	0.00%	7 Missing :warning:
...pache/cloudstack/api/response/ClusterResponse.java	0.00%	6 Missing :warning:
...g/apache/cloudstack/api/response/HostResponse.java	0.00%	6 Missing :warning:
... and 35 more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #9619 +/- ## ============================================ - Coverage 15.76% 15.76% -0.01% - Complexity 12507 12510 +3 ============================================ Files 5619 5621 +2 Lines 491158 491471 +313 Branches 62973 60171 -2802 ============================================ + Hits 77419 77460 +41 - Misses 405283 405555 +272 Partials 8456 8456 ``` | [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/9619/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/9619/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.04% <ø> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/9619/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `16.57% <15.05%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10893

nvazquez commented 2 months ago

@blueorangutan ui

blueorangutan commented 2 months ago

@nvazquez a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

UI build: :heavy_check_mark: Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9619 (QA-JID-436)

nvazquez commented 2 months ago

@blueorangutan test

blueorangutan commented 2 months ago

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan commented 2 months ago

[SF] Trillian test result (tid-11274) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 49550 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9619-t11274-kvm-ol8.zip Smoke tests completed. 139 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

rohityadavcloud commented 2 months ago

@nvazquez do we also need changes to deployment planner (orchestration), so VM deployment plan/planning considers arch of the iso/template against that of the cluster & hosts?

nvazquez commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10939

nvazquez commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10940

rohityadavcloud commented 2 months ago

@nvazquez can you merge latest main and address conflicts? Thanks.

nvazquez commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10956

nvazquez commented 2 months ago

@blueorangutan test

blueorangutan commented 2 months ago

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

github-actions[bot] commented 2 months ago

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions[bot] commented 2 months ago

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan commented 2 months ago

[SF] Trillian test result (tid-11346) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 53001 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9619-t11346-kvm-ol8.zip Smoke tests completed. 138 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_unsecure_vm_migration	`Error`	447.72	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	448.07	test_vm_life_cycle.py

rohityadavcloud commented 2 months ago

@nvazquez can you check/fix the lint issue & kick packaging. After that this should mergeable, pending no other issues.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10990

nvazquez commented 2 months ago

Thanks @rohityadavcloud - fixed

@blueorangutan package

blueorangutan commented 2 months ago

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

github-actions[bot] commented 2 months ago

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10997

DaanHoogland commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11012

rohityadavcloud commented 2 months ago

Minor conflict (no need for repackaging though), pl address @nvazquez

@blueorangutan test

blueorangutan commented 2 months ago

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

github-actions[bot] commented 2 months ago

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

rohityadavcloud commented 2 months ago

cc @nvazquez hope I haven't screwed up 🤞 the merge conflict

@blueorangutan package

blueorangutan commented 2 months ago

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11016

rohityadavcloud commented 2 months ago

@blueorangutan test

blueorangutan commented 2 months ago

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

rohityadavcloud commented 2 months ago

@blueorangutan package

blueorangutan commented 2 months ago

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented 2 months ago

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11024

blueorangutan commented 2 months ago

[SF] Trillian test result (tid-11386) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 55615 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9619-t11386-kvm-ol8.zip Smoke tests completed. 137 look OK, 1 have errors, 2 did not run Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_mount_shared_fs	`Failure`	93.24	test_sharedfs_lifecycle.py
test_resize_shared_fs	`Error`	0.23	test_sharedfs_lifecycle.py
all_test_network	`Skipped`	---	test_network.py
all_test_nic_adapter_type	`Skipped`	---	test_nic_adapter_type.py

apache / cloudstack