Tech debt for Chef Automate HA

[karunesh-tech]

Description:

This epic will primarily focus on all tech debt items related to automate HA. We will add a tech-debt label as well to all the issues associated with this epic.

List of stories/items in their priority order along with the issue link

#################FOR GA#################

• Automate HA to move to open search 1.0 - Done
  • Analyse the changes required for moving to open search
  • What if HA goes 1st with open search rather then Automate standalone
    • Check for the changes and challenges which it will inccure
  • Can only be done once the Automate has moved to open search - as will required necessary clients to be in place to communicate to external ES
• Automate HA - PG upgrade to PG 13 from PG 11 - Done
  • Test HA with the updated PG 13 package - Build and test
  • Add PG 13 package build and verify the service
  • List of all the possible testing we have to do
  • PG 9 to PG 13 (Automate to Automate HA)
  • PG 11 to PG 13 (A2HA to Automate HA)
  • Check above migration use cases
    • Verify Back up and restore for PG for all such scenarios
• Minimum node requirement - Done
• Performance benchmarking

Other tech debt:

• Support of -major flag on Automate HA upgrade - Done

• Better compatibility for frontend and backend upgrade commands in case of Semver versioning

• Integration with reporting manager

  • Will be done only once Automate is ready with the config driven external reporting manager support.
  • For AWS - provide a new machine provisioning and building with suggested mini HAB package
    • Terraform changes
    • Other config level checks in HA - to provide config driven provisioning
  • For On-prem - customer will enter the url for already provisioned mini setup and patch the config

• Trivy scanning for SAAS and Automate HA

  • Intergate with trivy tool and come up with scan result

• Testing for chef backend customer use case - Customer with very less number of node

  • Uses to be tested
    • Existing HA --> 10 nodes
    • Mini HA --> 5 nodes - - Done
    • Mini non-HA --> 4 nodes
    • Micro non-HA --> 2 nodes
    • Regular Automate - 1 node
  • Type of testing - benchmarking
    • Run in-general with default set retention period and check for concurrent achieved with base number of nodes
    • Run with only 1 day as retention and check for concurrent achieved with base number of nodes
  • This work will be done twice. Note will do just to test the feasibility
  • And once the performance issues are fixed by the Durga’s of team, we will test again for benchmarking.

• UX guidance for Automate command (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6405)

  • Work to be done in a feature branch and propose the same.
  • Based on the feedback proceed with the change. (UX - https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6235)

• Feedback from tecRacer on document- Content and sequencing - - Done

  • Share the master document with all the content to the tecRacer
  • Get the feedback on that on next. Call - 29th Dec - have doc team as well involved
  • Provide the feedback to doc team and collaborate together

• Support SSL with elastic search in Automate - - Done

  • Currently automate standalone do not have SSL for ES. And for external ES it is configured in a way to only support SSL based ES.
  • As part of this tech debt - we need to provide SSL in ES for standalone as well

• Automate HA to move to open search 1.0 - Done

  • Analyse the changes required for moving to open search
  • What if HA goes 1st with open search rather then Automate standalone
    • Check for the changes and challenges which it will inccure
  • Can only be done once the Automate has moved to open search - as will required necessary clients to be in place to communicate to external ES

• Automate HA - PG upgrade to PG 13 from PG 11 - Done

  • Test HA with the updated PG 13 package - Build and test
  • Add PG 13 package build and verify the service
  • List of all the possible testing we have to do
  • PG 9 to PG 13 (Automate to Automate HA)
  • PG 11 to PG 13 (A2HA to Automate HA)
  • Check above migration use cases
    • Verify Back up and restore for PG for all such scenarios

• Integration with reporting manager

  • Will be done only once Automate is ready with the config driven external reporting manager support.
  • For AWS - provide a new machine provisioning and building with suggested mini HAB package
    • Terraform changes
    • Other config level checks in HA - to provide config driven provisioning
  • For On-prem - customer will enter the url for already provisioned mini setup and patch the config

• Review the bugs for a2-backend

• Slack channel for a2ha_backend

• NO script usage in Automate HA - replace with commands - Done

  • Identify more such script driven approach
  • Replace with commands
    • Current doing for validation script

• Drop kibana, metrics beat, general beat - Done

• Move from public to private IP for ES in Automate HA - Done

• Test Automate HA on SUSE OS as well

• For e2e team - Integration testing for all the automate HA CLI commands - Automation of all the CLI commands verification

• Upgrade and Auto-upgrade testing

  • Manually BE upgrade testing
    • Manually ES and PG upgrade
  • Plan for POC/Spike for auto upgrades and approach

• Enhance cert rotation - Drop the script running and have it using command only

• Optimise the Airgap bundle

  • Current size of bundle is 1.9 GB approx, this story is to optimize the size of the bundle (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6304)

• Only relevant code to be pulled in bundle

  • Currently, the entire code base is in a single bundle (as part of automate deployment)
  • we should pull only those code that is required.

• Automate HA deployment should be done via automate deployment service

  • Currently we are calling automate cluster CLI deploy command internally to deploy automate HA.
  • Need to add a new service or changes in the existing one.
  • Ideally we should create new go lang service to pull the package and deploy Automate HA.

• Support non-Airgap (on the fly) type of installations.(Current working on it)

• Cover scenario for small scale customer:- Chef-Server and Automate running on a single VM

  • Planned testing in above if the tech debt item.

• Do not allow chef server services to run in Automate instances

• Managed service ( ES | PG ) support to Automate HA - Done

  • ES - https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6306
  • PG - https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6305

• Adding config like cloud details in yml to send logs to cloud watch.

  • We can add this as part of the documentation based on the config changes customer can integrate with AWS to get LMM support.

• Veracode scanning for automate HA (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6307)

• Advanced Validation lists (All type of validation should be done via automate ctl command only)

  • Pre-deployment validation command - Instances, server, ES, PG count (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6310)
  • Validation command can be ran at any point of time:- pre, post-deployment by the customer.
  • Verification on storage and HW/SW requirement (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6309)
  • Report generation for validation script (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6314)
  • CIDR validation (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6311)
  • Command addition to run validation script (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6312)

• Health check

  • PG metrics health check (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6316)
  • ES metrics - heap check and others (https://app.zenhub.com/workspaces/the-delta-quadrant-5fc7c4c3922dc10021323fa6/issues/chef/automate/6315)

• Sharing Configuration

• HA specific command should not be visible in CLI on automate non-HA.

• Versioning for Automate HA - backend components

• • Done Removing Kibana from the HA Arch | Should not send logs of PG and ES to same ES | Clean up the kibana, curator service.Public to private for ES ips

[btm]

HA specific command should not be visible in CLI on automate non-HA.

$ sudo chef-automate version
Version: 2
  CLI Build: 20211028055409
  Server Build: 20211102170523

$ sudo chef-automate --help | grep -i HA
  info                   InFo about automate HA
  init-config-ha         Initialize default config for HA
  provision-infra        Provison automate HA infra.
  secrets                set secrets to automate HA
  ssh                    Set Ssh env for automate HA.
  test                   Run automate HA smoke tests
  workspace              set workspace env for automate HA.

1. Can/should these all be moved under a top level "HA" command?
   • sudo chef-automate ha info
   • sudo chef-automate ha test
   • etc
2. Fix the descriptions
   • capitalize the first letter of each description, e.g. "Set secrets..."
   • capitalize Automate everywhere
   • remove the trailing period
   • Fix "InFo" to Info
   • Fix typo in Provison to Provision
3. do not display these commands if automate-cluster-ctl is not installed / check for dependencies

   
   $ sudo chef-automate info
   Automate HA info
   CommandExecutionError: An issue occurred when running an executable command: please refer

This command will be used to get information of all the components deployed for Automate HA . Usage: chef-automate info : exec: "automate-cluster-ctl": executable file not found in $PATH

4. should 'info' be merged into 'status'? can status simply show HA information if we're in an HA mode rather than adding a similar command?
5. should 'init-config-ha' be merged into 'init-config' ? can we just have `sudo chef-automate init-config --ha aws` ?