Multiple filters for Dynamic Catalog

[japneet-sahni]

Describe the bug When using multiple filters for creating dynamic host set plugin, only the last filter is taken into account. How do we actually get hosts populated based on multiple tags/filters (and condition)

To Reproduce Steps to reproduce the behavior:

1. Create 6 machines in Azure (3 windows and 3 linux)
2. Attach os:windows tag to windows machine and os:linux tag to linux machines
3. Attach team:dev to all 6 machines.
4. Create 2 dynamic host sets using above tags (one for windows and one for linux)

resource "boundary_host_catalog_plugin" "azure_catalog" {
  name        = "dynamic_azure_catalog"
  description = "Dynamic Azure host catalog"
  scope_id    = boundary_scope.project.id
  plugin_name = "azure"
  attributes_json = jsonencode({
    "disable_credential_rotation" = true,
    "tenant_id"                   = "${var.tenant_id}",
    "subscription_id"             = "${var.subscription_id}",
    "client_id"                   = "${var.client_id}"
  })
  secrets_json = jsonencode({
    "secret_value" = "${var.client_secret}"
  })
}

resource "boundary_host_set_plugin" "windows_servers" {
  name            = "Windows host set plugin"
  host_catalog_id = boundary_host_catalog_plugin.azure_catalog.id
  attributes_json = jsonencode({
    "filter" = "tagName eq 'team' and tagValue eq 'dev'",
    "filter" = "tagName eq 'os' and tagValue eq 'windows'",
  })
}

resource "boundary_host_set_plugin" "linux_servers" {
  name            = "Linux host set plugin"
  host_catalog_id = boundary_host_catalog_plugin.azure_catalog.id
  attributes_json = jsonencode({
    "filter" = "tagName eq 'team' and tagValue eq 'dev'",
    "filter" = "tagName eq 'os' and tagValue eq 'linux'",
  })
}

6. After Terraform apply the host sets are created but only the second filter (os tag) in attributes_json is taken into account (and not the team tag)

Expected behavior For windows host set, I should have hosts filtered on both tags (os:windows and team:dev) whereas, the hosts are actually filtered only on os tag i.e. os:windows

[xingluw]

Hi @japneet-sahni, this may be a limitation on Azure API but we are looking into this

[covetocove]

@japneet-sahni it seems like your requirement is to filter for hosts based off of multiple tagName values. Would it meet your requirement to define a single filter with multiple logical operators, rather than multiple distinct filters? An example of the above would be something like Single Filter: "filter" = ("tagName eq 'team' and tagValue eq 'dev'") and ("tagName eq 'os' and tagValue eq 'linux'") As opposed to multiple filters:

   "filter" = "tagName eq 'team' and tagValue eq 'dev'",
    "filter" = "tagName eq 'os' and tagValue eq 'linux'"

Defining multiple filters for a dynamic host would have negative performance repercussions. When requesting dynamic hosts from a catalog provider, Boundary acts as a client of the provider's api. Host set filters limit what resources are requested from the provider. Introducing multiple filters would lead to Boundary applying the filters on the client side rather than when making the request from the server (catalog provider), which is a recipe for requesting more hosts than are necessary and reducing performance.

Right now, our team believes there is a limitation in Azure's API that it does not support filters with multiple logical operations on the same field. We will work through this with Azure and share updates on progress.

[jefferai]

Hi @japneet-sahni ,

Azure doesn't support multiple filters in a call. We probably should raise an error if you try to specify multiple filters in the host-set configuration; right now the plugin simply ignores all but the last because it's a singular value. We are still going to be meeting with the MS team on ways forward though.