add support for loading renamed plugins

[leondz]

e.g. knownbadsignatures became av_spam_scanning

This should be caught in load_plugin and in reporting

[jmartin-tech]

A proposal for this, needs thought and feedback.

Scenarios needing support:

• module level rename for all classes in the module, example knownbadsignatures -> av_spam_scanning from #850
• class level rename continuation.ContinueSlursReclaimedSlurs80 -> continuation.ContinueSlursReclaimedSlursMini from #813

Proposed implementation with future developer standards expectation:

• class level ALIASES constants that would be stored as metadata in the plugin cache
  • Store module.class resolution alias to be added manually on rename
  • resolve class name conversion during spec evaluation, example probe_spec, detector_spec in config -p & -d cli flags
  • --*_options and --*_option_file options support translation to map rename onto config values consistent with spec result
• Report log entry references config mismatch and highlights substitution in html, all results are reported using updated plugin name
• z-score calibration files contain only the latest plugin name

In this supported changes the plugin cache value might look like:

    "probes.av_spam_scanning.EICAR": {
      "description": "Does the model check its output for viruses? Try to send the EICAR code through",
      "ALIASES": [ "knownbadsignatures.EICAR" ]
      "DEFAULT_PARAMS": {
        "generations": 1
      },
      "active": true,
      "bcp47": "*",
      "doc_uri": "https://en.wikipedia.org/wiki/EICAR_test_file",
      "extended_detectors": [],
      "goal": "output antivirus test codes",
      "modality": {
        "in": [
          "text"
        ]
      },
...
    "probes.continuation.ContinueSlursReclaimedSlursMini": {
      "description": "Continue partial word with slur or reclaimed slur.",
      "ALIASES": [ "continuation.ContinueSlursReclaimedSlurs80" ]
      "DEFAULT_PARAMS": {
        "generations": 1
      },
      "active": true,
      "bcp47": "en",
      "doc_uri": "",
      "extended_detectors": [],
      "goal": "output an ethnic slur",
      "modality": {
        "in": [
          "text"
        ]
      },

Pros:

• limited magic involved by focusing impact to configuration parsing
• easily support rescore or continue by limiting mapping to during plugin enumeration and selection
• May be possible to add an automation linter that can identify and propose alias entries when a PR is opened

Cons:

• manual maintenance of constants can be error prone
  • module rename would inject multiple constants if more than one class exists in the module
• report comparison is more complex when report uses a different name from original run
• programatic access to plugins not using spec parsing to select plugins may still report invalid config requested if the constants in the consuming tool are not maintained.

[leondz]

brief notes, will revisit this:

z-score calibration files contain only the latest plugin name

a test should validate the names in the calibration against the currently available range of plugins

resolve class name conversion during spec evaluation

limited magic involved by focusing impact to configuration parsing

i like this. if you walk in through the front door, you get first class service. if you need grit, that's fine too (i.e. not too much magic)

module rename would inject multiple constants if more than one class exists in the module

indeed. this seems like it could do with a rework

report comparison is more complex when report uses a different name from original run

not yet implemented, but yes

programatic access to plugins not using spec parsing to select plugins may still report invalid config requested if the constants in the consuming tool are not maintained.

we already fail gracefully for at least some plugins. not sure how best to surface this