refactor: blackbox testing

[friedrichwilken]

Description Because of the changes in test data the validation test does no longer work. This PR will changed this.

Changes proposed in this pull request:

• Handle a error where an expectation could not be assigned and would just be dropped.
• Data for evaluation and validation into one common directory per scenario;
  • validation data exists into ./data/namespace-scoped/<scenario name>/validation.yml
  • evaluation data exists into ./data/namespace-scoped/<scenario name>/scenario.yml
• Validation data is aggregated by evaluation scenario
• Added a unit test that ensures data integrity of validation test data.
• Added retry logic for validation test for situations where we run into rate limit issues. Related issue(s)

[github-actions[bot]]

The evaluation test will be skipped for this PR. You can trigger it manually by adding the label evaluation requested.

[github-actions[bot]]

The evaluation test will be skipped for this PR. You can trigger it manually by adding the label evaluation requested.

[github-actions[bot]]

The evaluation test will be skipped for this PR. You can trigger it manually by adding the label evaluation requested.

[friedrichwilken]

And it works:

[friedrichwilken]

The validation test now also handles rate limits with retries

2024-10-08 18:46:37 - INFO - validation.validator : Validating expectation: complete_yaml
2024-10-08 18:46:38 - WARNING - validation.validator : Rate limit error: Error code: 429 - {'error': 'TooManyRequest', 'message': 'Your request has been rate limited by AI Core. Please try again later.'}. Retrying in 2 minutes...
2024-10-08 18:48:39 - INFO - validation.validator : Validating expectation: deployable_yaml

[github-actions[bot]]

The evaluation test will be skipped for this PR. You can trigger it manually by adding the label evaluation requested.

[github-actions[bot]]

The evaluation test will be skipped for this PR. You can trigger it manually by adding the label evaluation requested.