Rewrite of the descriptor-api to improve usability for developers

[kKdH]

In my opinion, writing descriptors for sources, filters and sinks is a pain. There are multiple reasons why we should rewrite the descriptor-api.

Problem 1: Mix of parameter definition and localization

Mixing the parameter definition an the translation of the name and description pollutes the code and make it hard to read. Additionally creating a map of descriptors (to support multiple languages) makes it even worse. In the example below taken from the AddFieldsFilterLogic is a lot of code related to localization and not defining parameters:

private val filterName = "io.logbee.keyscore.agent.pipeline.contrib.filter.AddFieldsFilterLogic"
private val bundleName = "io.logbee.keyscore.agent.pipeline.contrib.filter.AddFieldsFilter"
private val filterId = "1a6e5fd0-a21b-4056-8a4a-399e3b4e7610"

override def describe: MetaFilterDescriptor = {
  val descriptorMap = mutable.Map.empty[Locale, FilterDescriptorFragment]
  descriptorMap ++= Map(
    Locale.ENGLISH -> descriptor(Locale.ENGLISH),
    Locale.GERMAN -> descriptor(Locale.GERMAN)
  )
  MetaFilterDescriptor(fromString(filterId), filterName, descriptorMap.toMap)
}

private def descriptor(language: Locale): FilterDescriptorFragment = {
  val translatedText: ResourceBundle = ResourceBundle.getBundle(bundleName, language)
  FilterDescriptorFragment(
    displayName = translatedText.getString("displayName"),
    description = translatedText.getString("description"),
    previousConnection = FilterConnection(true),
    nextConnection = FilterConnection(true),
    parameters = List(
      MapParameterDescriptor("fieldsToAdd", translatedText.getString("fieldsToAddName"), translatedText.getString("fieldsToAddDescription"),
        TextParameterDescriptor("fieldName", translatedText.getString("fieldKeyName"), translatedText.getString("fieldKeyDescription")),
        TextParameterDescriptor("fieldValue", translatedText.getString("fieldValueName"), translatedText.getString("fieldValueDescription"))
      )
    ))
}

Problem 2: Extraction of parameter values from a given configuration

To extract the parameter values from a given configuration i have to search through the configuration and check the parameter by its name given as string. This approach is error prone and needs a lot boiler plate code as shown in the Example below also taken from the AddFieldsFilterLogic:

for (parameter <- configuration.parameters) {
  parameter.name match {
    case "fieldsToAdd" =>
      val dataMap = parameter.value.asInstanceOf[Map[String, String]]
      dataToAdd ++= dataMap.map(pair => (pair._1, TextField(pair._1, pair._2)))
    case _ =>
  }
}

Problem 3: Automatic configuration checking

There is no automatic check whether a given configuration suits a certain logic. The developer of a filter has to write it on its own or the filter crashes at runtime due to a missing paramter.

Suggestions

• It should be possible to translate categories.
• Greater variety of ParameterDescriptors:
  • e.g. a separate descriptor for:
    • regular expressions (patterns)
    • numbers with unit and precision information.
    • field names with information about whether they have to be present or absent in a dataset.
  • The UI could use these additional information to render specialized ui-controls which would in turn enhance the user experience.

Proposal

Scala Example

import io.logbee.keyscore.model.FieldNameHint.PresentField
import io.logbee.keyscore.model.PatternType.Grok
import io.logbee.keyscore.model._
import io.logbee.keyscore.model.localization.{Locale, Localization, TextRef}

val EN = Locale("en", "US")
val DE = Locale("de/DE")

val filterDisplayName = TextRef("displayName")
val filterDescription = TextRef("description")
val category = TextRef("aa5de1cd-1122-758f-97fa-228ca8911378")

val parameterARef = ParameterRef("37024d8b-4aec-4b3e-8074-21ef065e5ee2")
val parameterADisplayName = TextRef("parameterADisplayName")
val parameterADescription = TextRef("parameterADescription")

//      val parameterBRef = ParameterRef("ff543cab-15bf-114a-47a1-ce1f065e5513")
val parameterBDisplayName = TextRef("parameterBDisplayName")
val parameterBDescription = TextRef("parameterBDescription")

val parameterCRef = ParameterRef("b7cc9c84-ae6e-4ea3-bbff-f8d62af4caed")
//      val parameterDRef = ParameterRef("5f28c6dd-f88f-4530-afd1-c8b946bc5406")

val descriptor = Descriptor(
id = "1a6e5fd0-a21b-4056-8a4a-399e3b4e7610",
describe = FilterDescriptor(
  name = "io.logbee.keyscore.agent.pipeline.contrib.filter.AddFieldsFilterLogic",
  displayName = filterDisplayName,
  description = filterDescription,
  category = category,
  parameters = Seq(
    TextParameterDescriptor(parameterARef, ParameterInfo(parameterADisplayName, parameterADescription), defaultValue = "Hello World", validator = StringValidator("Hello*", PatternType.Glob)),
    BooleanParameterDescriptor(parameterCRef, ParameterInfo(TextRef("parameterDDisplayName"), TextRef("parameterDDescription")), defaultValue = true),
    ConditionalParameterDescriptor(condition = BooleanParameterCondition(parameterCRef, negate = true), parameters = Seq(
      PatternParameterDescriptor("98276284-a309-4f21-a0d8-50ce20e3376a", patternType = Grok),
      ListParameterDescriptor("ff543cab-15bf-114a-47a1-ce1f065e5513",
        ParameterInfo(parameterBDisplayName, parameterBDescription),
        FieldNameParameterDescriptor(hint = PresentField, validator = StringValidator("^_.*", PatternType.RegEx)),
        min = 1, max = Int.MaxValue)
    )),
    ChoiceParameterDescriptor("e84ad685-b7ad-421e-80b4-d12e5ca2b4ff", min = 1, max = 1, choices = Seq(
      Choice("red"),
      Choice("green"),
      Choice("blue")
    ))
  )
),
localization = Localization(Set(EN, DE), Map(
  filterDisplayName -> Map(
    EN -> "Add Fields",
    DE -> "Feld Hinzufuegen"
  ),
  filterDescription -> Map(
    EN -> "Adds the specified fields.",
    DE -> "Fuegt die definierten Felder hinzu."
  ),
  category -> Map(
    EN -> "Source",
    DE -> "Quelle"
  ),
  parameterADisplayName -> Map(
    EN -> "A Parameter",
    DE -> "Ein Parameter"
  ),
  parameterADescription -> Map(
    EN -> "A simple text parameter as example.",
    DE -> "Ein einfacher Textparameter als Beispiel."
  ),
  parameterBDisplayName -> Map(
    EN -> "A Parameter",
    DE -> "Ein Parameter"
  ),
  parameterBDescription -> Map(
    EN -> "A simple text parameter as example.",
    DE -> "Ein einfacher Textparameter als Beispiel."
  )
)
))

JSON Output

{
  "jsonClass": "io.logbee.keyscore.model.Descriptor",
  "id": "1a6e5fd0-a21b-4056-8a4a-399e3b4e7610",
  "describe": {
    "jsonClass": "io.logbee.keyscore.model.FilterDescriptor",
    "name": "io.logbee.keyscore.agent.pipeline.contrib.filter.AddFieldsFilterLogic",
    "displayName": {
      "id": "displayName"
    },
    "description": {
      "id": "description"
    },
    "category": {
      "id": "aa5de1cd-1122-758f-97fa-228ca8911378"
    },
    "parameters": [
      {
        "jsonClass": "io.logbee.keyscore.model.TextParameterDescriptor",
        "ref": {
          "id": "37024d8b-4aec-4b3e-8074-21ef065e5ee2"
        },
        "info": {
          "displayName": {
            "id": "parameterADisplayName"
          },
          "description": {
            "id": "parameterADescription"
          }
        },
        "defaultValue": "Hello World",
        "validator": {
          "pattern": "Hello*",
          "patternType": "Glob"
        }
      },
      {
        "jsonClass": "io.logbee.keyscore.model.BooleanParameterDescriptor",
        "ref": {
          "id": "b7cc9c84-ae6e-4ea3-bbff-f8d62af4caed"
        },
        "info": {
          "displayName": {
            "id": "parameterDDisplayName"
          },
          "description": {
            "id": "parameterDDescription"
          }
        },
        "defaultValue": true
      },
      {
        "jsonClass": "io.logbee.keyscore.model.ConditionalParameterDescriptor",
        "ref": {
          "id": ""
        },
        "condition": {
          "jsonClass": "io.logbee.keyscore.model.BooleanParameterCondition",
          "parameter": {
            "id": "b7cc9c84-ae6e-4ea3-bbff-f8d62af4caed"
          },
          "negate": true
        },
        "parameters": [
          {
            "jsonClass": "io.logbee.keyscore.model.PatternParameterDescriptor",
            "ref": {
              "id": "98276284-a309-4f21-a0d8-50ce20e3376a"
            },
            "patternType": "Grok",
            "defaultValue": ""
          },
          {
            "jsonClass": "io.logbee.keyscore.model.ListParameterDescriptor",
            "ref": {
              "id": "ff543cab-15bf-114a-47a1-ce1f065e5513"
            },
            "info": {
              "displayName": {
                "id": "parameterBDisplayName"
              },
              "description": {
                "id": "parameterBDescription"
              }
            },
            "kind": {
              "jsonClass": "io.logbee.keyscore.model.FieldNameParameterDescriptor",
              "ref": {
                "id": ""
              },
              "defaultValue": "",
              "hint": "PresentField",
              "validator": {
                "pattern": "^_.*",
                "patternType": "RegEx"
              }
            },
            "min": 1,
            "max": 2147483647
          }
        ]
      },
      {
        "jsonClass": "io.logbee.keyscore.model.ChoiceParameterDescriptor",
        "ref": {
          "id": "e84ad685-b7ad-421e-80b4-d12e5ca2b4ff"
        },
        "min": 1,
        "max": 1,
        "choices": [
          {
            "name": "red"
          },
          {
            "name": "green"
          },
          {
            "name": "blue"
          }
        ]
      }
    ]
  },
  "localization": {
    "locales": [
      {
        "language": "en",
        "country": "US"
      },
      {
        "language": "de",
        "country": "DE"
      }
    ],
    "mapping": {
      "parameterADisplayName": {
        "translations": {
          "en/US": "A Parameter",
          "de/DE": "Ein Parameter"
        }
      },
      "description": {
        "translations": {
          "en/US": "Adds the specified fields.",
          "de/DE": "Fuegt die definierten Felder hinzu."
        }
      },
      "parameterBDisplayName": {
        "translations": {
          "en/US": "A Parameter",
          "de/DE": "Ein Parameter"
        }
      },
      "aa5de1cd-1122-758f-97fa-228ca8911378": {
        "translations": {
          "en/US": "Source",
          "de/DE": "Quelle"
        }
      },
      "parameterADescription": {
        "translations": {
          "en/US": "A simple text parameter as example.",
          "de/DE": "Ein einfacher Textparameter als Beispiel."
        }
      },
      "displayName": {
        "translations": {
          "en/US": "Add Fields",
          "de/DE": "Feld Hinzufuegen"
        }
      },
      "parameterBDescription": {
        "translations": {
          "en/US": "A simple text parameter as example.",
          "de/DE": "Ein einfacher Textparameter als Beispiel."
        }
      }
    }
  }
}

[MaxKarthan]

As part of rewriting the descriptor api we should keep in mind that at the moment the category flag has no display name. We are not able to translate the category as it is now. Not a big deal, just to keep in mind´

I think it would also be great to offer the possibility to group categories. I don't think that the groups would need names. Simply putting categories in one list or so to group them. This would give us the opportunity to make a simpler, cleaner and more organized UI .

[kKdH]

Good point! With the new API it should be possible to translate categories too.

I don’t understand your suggestion about grouping categories. Can you give an example?

[kKdH]

I'm going to close this issue for clarity of milestone progress.