Documentation: Add the list of tags available for the `only` param of the `scan` method

[kevinmessiaen]

🚀 Feature Request

Add the list of tags available for the only param of the scan method in the documentation.

🔈 Motivation

The only parameter accept a List[str] but it is undocumented. Therefore to know what tags to put we have to investigate the detectors source code.

🛰 Alternatives

Add a method to dynamically list the detectors and tags in Giskard: giskard.list_detectors()

📎 Additional context

Detectors are registered using the @detector decorator see here: https://github.com/Giskard-AI/giskard/blob/f68a495ef1df470d55790fa7c2308303881e30d6/giskard/scanner/llm/llm_chars_injection_detector.py#L14

The tags parameters allow to only run specific detectors when using giskard.scan(model, dataset, only=['list', 'of', 'tags']), the rule is to run only the detectors that contains at least one tag included in the only list.

[gopi-nath]

Any update on this documentation? Much needed!

[Prince97123]

You have updated the documentation mentioning the tag name in comments for the detector value to be passed in the "only" param.

I tried with a few detectors but it gave the error: *No issue detector available. The scan will not be performed. for e.g. gsk.scan(model, demo_dataset, only=["text_perturbation"])

Please check this.

[Prince97123]

Here we are using the latest giskard version and when tried with "perfomance_bias" detector it's giving error. Please have a look

sh-5.2$ pip3 freeze|grep giskard
giskard==2.15.1

python3 test_chatbot_without_input.py
2024-09-25 12:33:42,113 pid:2701485 MainThread giskard.models.automodel INFO     Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
2024-09-25 12:33:42,119 pid:2701485 MainThread giskard.datasets.base INFO     Your 'pandas.DataFrame' is successfully wrapped by Giskard's 'Dataset' wrapper class.
🔎 Running scan…
Estimated calls to your model: ~0
Estimated LLM calls for evaluation: 0

Traceback (most recent call last):
  File "/home/ssm-user/6_sep/test_chatbot_without_input.py", line 120, in <module>
    scan_results = gsk.scan(model, demo_dataset, only=["performance_bias"])
  File "/home/ssm-user/.local/lib/python3.9/site-packages/giskard/scanner/__init__.py", line 67, in scan
    return scanner.analyze(
  File "/home/ssm-user/.local/lib/python3.9/site-packages/giskard/scanner/scanner.py", line 126, in analyze
    issues, errors = self._run_detectors(
  File "/home/ssm-user/.local/lib/python3.9/site-packages/giskard/scanner/scanner.py", line 152, in _run_detectors
    raise RuntimeError("No issue detectors available. Scan will not be performed.")
RuntimeError: No issue detectors available. Scan will not be performed.

[kevinmessiaen]

@Prince97123 Hello, the "perfomance_bias" detector is a detector that is only applicable to classification and regression models. Hence it's not found for LLM detectors.

But I agree with you that we can probably improve the error saying that it's found but not applicable.

[Prince97123]

Hey,

Please let me know which detectors are available for model_type as text_generation? So far, I have only been able to find the Jailbreak and Faithfulness detectors for text generation models.

If I want to scan my model for other vulnerabilities, what options or approaches would you recommend? Are there additional detectors available?

Thanks