aiverify-foundation / moonshot-data

Contains all assets to run with Moonshot Library (Connectors, Datasets and Metrics)
Apache License 2.0
18 stars 17 forks source link

[MS-292] Removal of generating dynamic benchmark prompts on the fly #71

Closed imda-lionelteo closed 3 months ago

imda-lionelteo commented 3 months ago

Description

Removal of generating dynamic benchmark prompts on the fly

Motivation and Context

This is to remove the feature where we will generate dynamic benchmark for dataset prompts on the fly. This feature will be replaced with augmented benchmark where it will create a new recipe with a new dataset that contains the modified prompts.

Type of Change

How to Test

Perform a run_recipe and run_cookbook to get the benchmark to run. There are no more attack modules that will augment the prompts anymore.

Checklist

Please check all the boxes that apply to this pull request using "x":

Screenshots (if applicable)

[If the changes involve visual modifications, include screenshots or GIFs that demonstrate the changes.]

Additional Notes

There is an accompanying update in moonshot

Developer Certificate of Origin ``` Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ```
imda-benedictlee commented 3 months ago

Link to Jira Ticket: https://imda-dsl.atlassian.net/browse/MS-292?atlOrigin=eyJpIjoiMTg4M2VkZjk0MzAyNDBjZGEyNTJkNjRhMzY2M2I1NWQiLCJwIjoiaiJ9

imda-benedictlee commented 3 months ago

Tested. Working as expected.

Screenshot 2024-07-31 at 4 59 32 PM