Add list-suts command

[brianwgoldman]

Similar to list-tests, we should have a list-suts command that as a minimum:

1. List all the valid values for --sut.
2. If a SUT requires secrets, list those secrets. Maybe condition this to only listing secrets you don't already have set.

[0xkerem]

Hello @brianwgoldman ! I've looked into this a little bit, but I have a question for you. There is no get_metadata() method in the demo_plugin\newhelm\suts\demo_01_yes_no_sut.py example. In this case, does list_suts() mean that it should not be displayed in the meta or will it be implemented later? If it is not available, can you give brief details about it? So nothing will appear on the screen except the Command line key?

[brianwgoldman]

There is no get_metadata() method

We don't have a great story yet for metadata in general. That method in BaseTest is kind of a placeholder until we figure it all out. There isn't a corresponding one for SUT because it felt like there was more of a detachment between the SUT class and its instances than between the Test class and its instances.

I don't think this is something we need to resolve in this issue though.

If it is not available, can you give brief details about it? So nothing will appear on the screen except the Command line key?

I was imagining something like this:

llama-2-7b
class=TogetherCompletionsSUT
args=('togethercomputer/llama-2-7b', ...)
kwargs={}
Missing Required secrets:
scope="together", key="api_key", instructions="See https://api.together.xyz/settings/api-keys"

We could have that last block also have Uses instead of Missing if you have a value for it.

Somewhat related, we should probably give better formatting to Inject. For example in the current list command:

llama-2-7b TogetherCompletionsSUT(args=('togethercomputer/llama-2-7b', <newhelm.secret_values.InjectSecret object at 0x79e8ca2167a0>), kwargs={})

That should probably be something like

llama-2-7b TogetherCompletionsSUT(args=('togethercomputer/llama-2-7b', Inject(TogetherApiKey), kwargs={})

[0xkerem]

Thank you for explaining it very well in detail. This is what it looks like in its current state. I haven't created Inject's repr yet, but it looks like this with string operations. For now, I need to sleep because it is very late here :) , but I will continue in the morning according to your feedback.

$ poetry run python newhelm/main.py list-suts
Cannot display SUT llama-2-7b because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT llama-2-70b because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT falcon-40b because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT llama-2-13b because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT flan-t5-xl because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT llama-2-7b-chat because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT llama-2-70b-chat because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT zephyr-7b-beta because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT vicuna-13b-v1.5 because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT Mistral-7B-Instruct-v0.2 because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT WizardLM-13B-V1.2 because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT oasst-sft-4-pythia-12b-epoch-3.5 because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT dolly-v2-12b because it requires the following secrets:
scope='together' key='api_key' instructions='See https://api.together.xyz/settings/api-keys'

Cannot display SUT gpt-3.5-turbo because it requires the following secrets:
scope='openai' key='api_key' instructions='See https://platform.openai.com/api-keys'

demo_yes_no
class=DemoYesNoSUT
args=[]
kwargs={}
Uses required secrets:
{'demo': {'demo_api_key': '12345'}}

gpt2
class=HuggingFaceSUT
args=['gpt2', SerializedSecret(module='newhelm.suts.huggingface_client', qual_name='HuggingFaceToken')]
kwargs={}
Uses required secrets:
{'demo': {'demo_api_key': '12345'}}

[brianwgoldman]

Fixed by #263