Slight Change in Management Group Hierarchy

khabbabs commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Not sure if this is an a feature request but I do need some help in figuring out if something like this is possible. I'm trying to keep the recommended hierarchy but with an additional layer. Basically keeping the core landing zones but with another layer added.

Something like

Ignore the nonprod management group. I want to take the core management connectivity and identity groups under another layer. Really hope the pic helps describe what i'm wanting.

Is your feature request related to a problem?

I can create a similar hierarchy using the custom landing zones but I can't deploy all resources for it that come with the core management groups.

Any suggestions would be appreciated.

Describe the solution you'd like

I would like to create a custom landing zone under the parent management group like ...

"${local.root_id}-management-prod" = {
      display_name               = "prod"
      parent_management_group_id = "${local.root_id}-management"
      subscription_ids           = ["${var.subscription_id_management}"]
      archetype_config = {
        archetype_id   = "default_empty"
        parameters     = {}
        access_control = {}
      }
    }

at the same time I would like to deploy the resources generated from settings.mangement.tf

locals {
  configure_management_resources = {
    settings = {
      log_analytics = {
        enabled = true
        config = {
          retention_in_days                           = 396
          enable_monitoring_for_arc                   = false
          enable_monitoring_for_vm                    = true
          enable_monitoring_for_vmss                  = true
          enable_solution_for_agent_health_assessment = true
          enable_solution_for_anti_malware            = true
          enable_solution_for_azure_activity          = true
          enable_solution_for_change_tracking         = true
          enable_solution_for_service_map             = true
          enable_solution_for_sql_assessment          = true
          enable_solution_for_updates                 = true
          enable_solution_for_vm_insights             = true
          enable_sentinel                             = false
        }
      }
      security_center = {
        enabled = true
        config = {
          email_security_contact             = "InformationSecurity@YOLO.com"
          enable_defender_for_acr            = false
          enable_defender_for_app_services   = true
          enable_defender_for_arm            = false
          enable_defender_for_dns            = true
          enable_defender_for_key_vault      = false
          enable_defender_for_kubernetes     = false
          enable_defender_for_servers        = true
          enable_defender_for_sql_servers    = true
          enable_defender_for_sql_server_vms = true
          enable_defender_for_storage        = true
        }
      }
    }

    location = null
    tags = {
      deployedBy = "terraform/azure/caf-enterprise-scale"
    }
    advanced = {
      # resource_prefix = "${var.envtype}-${var.location}"
      resource_prefix = var.env
      resource_suffix = var.location

      custom_settings_by_resource_type = {
        azurerm_log_analytics_workspace = {
          management = {
            name = "${var.env}-${var.location}-la-01"
          }
        }
      }
    }
  }
}

the main file has this for the management resource

deploy_management_resources    = true
configure_management_resources = local.configure_management_resources
subscription_id_management     = data.azurerm_client_config.management.subscription_id

This keeps the subscription under the parent management group and no the prod management group that I created in the custom landing zones section.

Please let me know if you need any more clarification! Really love the solution you already have, I would just like to make some small adjustments.

Additional context

TBD

krowlandson commented 2 years ago

Hi @khabbabs... thank you for this interesting request. We've actually discussed similar scenarios with other customers and already have an answer for this, although we should probably make this clearer in the broader Enterprise-scale documentation.

In short, the design was intentionally meant to have a 1:1 relationship between each of the platform Management Groups and their corresponding Subscriptions due to the way some policies were written. I'm not sure if this is so critical now (this was mainly relating to connectivity policies with IPAM parameters which have since been removed) but we still recommend that you do not use Management Groups to differentiate between environments.

That said, if you really want to do something like this, whilst retaining functionality of the management resources you will need to ensure your management Subscription lands in your management Management Group as this is how the module ensures all the resources are in the correct place.

Although I wouldn't recommend this, to achieve your desired hierarchy you can use the following as a baseline for your module config:

module "enterprise_scale" {
  source  = "Azure/caf-enterprise-scale/azurerm"
  version = "1.0.0"

  providers = {
    azurerm              = azurerm.management
    azurerm.connectivity = azurerm.management
    azurerm.management   = azurerm.management
  }

  root_parent_id = data.azurerm_client_config.management.tenant_id
  root_id        = local.root_id
  root_name      = local.root_name

  custom_landing_zones = {
    # Create a new base Management Group under platform for `Management`
    # Must have a different ID such as below
    "${local.root_id}-management-base" = {
      display_name               = "Management"
      parent_management_group_id = "${local.root_id}-platform"
      subscription_ids           = []
      archetype_config = {
        # Map this to the default_empty archetype definition
        archetype_id   = "default_empty"
        parameters     = {}
        access_control = {}
      }
    }
    # Redefine the built-in `management` Management Group
    # Set the display_name to your desired value of `prod`
    # Set the parent_management_group_id to the ID of the MG above
    "${local.root_id}-management" = {
      display_name               = local.env # I assume this maps correctly to your desired value of `prod`
      parent_management_group_id = "${local.root_id}-management-base"
      subscription_ids           = [
        # Need to manually map management.subscription_id as the built-in logic is overriden by this config
        data.azurerm_client_config.management.subscription_id,
      ]
      archetype_config = {
        # Make sure you map this to the original es_management archetype definition
        archetype_id   = "es_management"
        parameters     = {}
        access_control = {}
      }
    }
  }

  deploy_management_resources    = true
  configure_management_resources = local.configure_management_resources
  subscription_id_management     = data.azurerm_client_config.management.subscription_id

}

This should give you the desired structure as below, whilst minimising the impact on integration functionality.

Please let me know how you get on!

jtracey93 commented 2 years ago

Also just to chip in here. I would recommend a review of https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/enterprise-scale/testing-approach that may be more suited to your requirements and change the hierarchy 👍

amhcloud commented 2 years ago

Do you still need to set deploy_core_landing_zones as "true", or does this need to be changed to false to deploy it this way Kevin?

amhcloud commented 2 years ago

Also just to chip in here. I would recommend a review of https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/enterprise-scale/testing-approach that may be more suited to your requirements and change the hierarchy 👍

Thanks for this. How do you ensure logs from the canary still get distributed to the primary log analytics workspace, instead of provisioning new for, "Send all Azure activity logs for all Azure subscriptions, including any canary environment subscriptions, to the production environment Azure Log Analytics workspace as per the enterprise-scale design recommendations."

krowlandson commented 2 years ago

Do you still need to set deploy_core_landing_zones as "true", or does this need to be changed to false to deploy it this way Kevin?

In the example above, the assumption is to have deploy_core_landing_zones = true which is the default value anyway.

If you don't want the remaining hierarchy you can set this to false but will need to build the entire hierarchy within custom_landing_zones

krowlandson commented 2 years ago

Also just to chip in here. I would recommend a review of https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/enterprise-scale/testing-approach that may be more suited to your requirements and change the hierarchy 👍

Thanks for this. How do you ensure logs from the canary still get distributed to the primary log analytics workspace, instead of provisioning new for, "Send all Azure activity logs for all Azure subscriptions, including any canary environment subscriptions, to the production environment Azure Log Analytics workspace as per the enterprise-scale design recommendations."

Great question... we'll come back to you on this.

cc: @jtracey93

jtracey93 commented 2 years ago

Also just to chip in here. I would recommend a review of https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/enterprise-scale/testing-approach that may be more suited to your requirements and change the hierarchy 👍

Thanks for this. How do you ensure logs from the canary still get distributed to the primary log analytics workspace, instead of provisioning new for, "Send all Azure activity logs for all Azure subscriptions, including any canary environment subscriptions, to the production environment Azure Log Analytics workspace as per the enterprise-scale design recommendations."

Great question... we'll come back to you on this.

cc: @jtracey93

In terms of this module (@krowlandson correct me if I'm wrong here 👍) you would need to just ensure the parameter value passed into the policy assignment in your canary deployment for Deploy-AzActivity-Log is set to the "production" log analytics workspace. You can do this with an archetype_config_overrides as below:

archetype_config_overrides = {
    root = {
        archetype_id   = "es_root"
        parameters     = {
          Deploy-AzActivity-Log = {
            logAnalytics = "/subscriptions/11111111-1111-1111-1111-000000000000/resourcegroups/test-mgmt/providers/Microsoft.OperationalInsights/workspaces/test-la"
          }
        }
        access_control = {}
    }
  }

Replace the resource ID with your own

You need to ensure the following:

The root_id is unique between prod and canary environments
Subscriptions used in prod and canary environments are different constantly to avoid conflict between the environments
Use the archetype_config_overrides as above to change the input value for the Deploy-AzActivity-Log policy in your canary environment

khabbabs commented 2 years ago

Hey @krowlandson in your example how did you make sure it only deploys one management group? I followed your example and i'm getting two different management groups, one from the custom landing zones "myorg-management-base" and another which is part of the core landing zones "myorg-management".

Here's the custom_landing_zones code

 custom_landing_zones = local.deploy_custom_landing_zones ? {
    "${local.root_id}-management-base" = {
      display_name               = "Management"
      parent_management_group_id = "${local.root_id}-platform"
      subscription_ids           = []
      archetype_config = {
        archetype_id   = "default_empty"
        parameters     = {}
        access_control = {}
      }
    }
    "${local.root_id}-management-prod" = {
      display_name               = "prod"
      parent_management_group_id = "${local.root_id}-management-base"
      subscription_ids           = [
        data.azurerm_client_config.management.subscription_id,
        ]
      archetype_config = {
        archetype_id   = "es_management"
        parameters     = {}
        access_control = {}
      }
    }
    "${local.root_id}-management-nonprod" = {
      display_name               = "nonprod"
      parent_management_group_id = "${local.root_id}-management-base"
      subscription_ids           = []
      archetype_config = {
        archetype_id   = "es_management"
        parameters     = {}
        access_control = {}
      }
    }

  deploy_management_resources        = true
  configure_management_resources   = local.configure_management_resources
  subscription_id_management            = data.azurerm_client_config.management.subscription_id

jtracey93 commented 2 years ago

Hey @krowlandson in your example how did you make sure it only deploys one management group? I followed your example and i'm getting two different management groups, one from the custom landing zones "myorg-management-base" and another which is part of the core landing zones "myorg-management".

Here's the custom_landing_zones code


 custom_landing_zones = local.deploy_custom_landing_zones ? {

    "${local.root_id}-management-base" = {

      display_name               = "Management"

      parent_management_group_id = "${local.root_id}-platform"

      subscription_ids           = []

      archetype_config = {

        archetype_id   = "default_empty"

        parameters     = {}

        access_control = {}

      }

    }

    "${local.root_id}-management-prod" = {

      display_name               = "prod"

      parent_management_group_id = "${local.root_id}-management-base"

      subscription_ids           = [

        data.azurerm_client_config.management.subscription_id,

        ]

      archetype_config = {

        archetype_id   = "es_management"

        parameters     = {}

        access_control = {}

      }

    }

    "${local.root_id}-management-nonprod" = {

      display_name               = "nonprod"

      parent_management_group_id = "${local.root_id}-management-base"

      subscription_ids           = []

      archetype_config = {

        archetype_id   = "es_management"

        parameters     = {}

        access_control = {}

      }

    }

  deploy_management_resources        = true

  configure_management_resources   = local.configure_management_resources

  subscription_id_management            = data.azurerm_client_config.management.subscription_id

Hey @khabbabs I think this comment covers this off https://github.com/Azure/terraform-azurerm-caf-enterprise-scale/issues/200#issuecomment-962022810

👍

krowlandson commented 2 years ago

Hey @krowlandson Kevin Rowlandson FTE in your example how did you make sure it only deploys one management group? I followed your example and i'm getting two different management groups, one from the custom landing zones "myorg-management-base" and another which is part of the core landing zones "myorg-management".

The critical part here is re-defining the built-in Management Group definition for ${local.root_id}-management. In your example, you and a suffix to all of the Management Group IDs, so the default one will still be deployed and then used as the target for all module logic.

This means you would effectively have the following management Management Groups:

ID	Display Name	Parent
`${local.root_id}-management-base`	`Management`	`${local.root_id}-platform`
`${local.root_id}-management`	`prod`	`${local.root_id}-management-base`
`${local.root_id}-management-nonprod`	`nonprod`	`${local.root_id}-management-base`

As an aside, it would be interesting to understand your use-case as environment aligned Management Groups is actually an ES anti-pattern. We also recommend only implementing a single Log Analytics workspace for all logs. It will be very challenging to correctly configure Policy Assignments to ensure things like Activity Logs and Diagnostics go to the correct Log Analytics workspace, but using multiple Log Analytics workspaces will also greatly increase the complexities associated with running queries across all Subscriptions and integration with external tooling (e.g. SIEM, etc.)

khabbabs commented 2 years ago

@krowlandson and @jtracey93 thank you so much for all your help and suggestions. At this moment we're still experimenting with what we want to do.

I had another question regarding custom landing zones. How does the arche type exclusion work with custom landing zones?

for example.

I have a file archetype_exclusion_es_identity.tmpl.json

{
    "exclude_es_identity": {
      "policy_assignments": [
        "Deny-Subnet-Without-NSG"
      ],
      "policy_definitions": [],
      "policy_set_definitions": [],
      "role_definitions": [],
      "archetype_config": {
        "parameters": {},
        "access_control": {}
      }
    }
  }

Does this exclusion get inherited to the custom landing zones which have "es_identity" listed in the archetype_id...

 "${local.root_id}-identity-prod" = {
      display_name               = "prod"
      parent_management_group_id = "${local.root_id}-identity"
      subscription_ids           = ["${var.subscription_id_identity}"]
      archetype_config = {
        archetype_id   = "es_identity"
        parameters     = {}
        access_control = {}
      }
    }

jtracey93 commented 2 years ago

Hey @khabbabs,

The exclusion feature works against the archetype definitions themselves, so are agnostic to custom landing zones etc.

So you may find the case where if you need to different exclusion from the same archetype definition on different management groups, you may actually need to create separate archetype definitions to achieve this 👍 (which aligns with archetypes - as they will be different 😃)

Hope that helps

Jack

khabbabs commented 2 years ago

Thanks for all your help! if we have any more questions, i'll review the docs before opening up another issue if need be.

Azure / terraform-azurerm-caf-enterprise-scale

Slight Change in Management Group Hierarchy #200

Community Note

Description