alat-rights / alana-utilities

Quick utilities for interacting with LLMs.
https://utils.alana.computer
MIT License
1 stars 2 forks source link

Cannot perform unittest without valid key #10

Open Domejko opened 5 months ago

Domejko commented 5 months ago

While I wanted to perform tests I did encounter this error on 12 from 16 test cases:

TypeError: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the `X-Api-Key` or `Authorization` headers to be explicitly omitted"

Test could not be performed without Claude account with resources on it. I think that for general testing purposes we will need to mock Anthropic client.messages.create.

alat-rights commented 5 months ago

Hi!

Yes, tests are currently run live via the Anthropic API.

While I'm open to mocking, this is not currently considered a bug.

Can you get yourself an Anthropic API key here, and then add os.environ["ANTHROPIC_API_KEY"] = your_api_key to your local copy of the test code?

Definitely want to automate this with GitHub Actions in the future.

Domejko commented 5 months ago

I have already set up an account and used valid generated api_key but then it was demanding to top up my credits, there is a option go get 5$ for free but I'm not convinced to give my phone number left and right.

I'll will try to mock that return value so that testing will be independent of valid api_key.

alat-rights commented 5 months ago

Ah, I see!

I think a graceful way to handle this would be to keep the existing functions as-is and add new ones with mocked outputs.

Proposal:

Domejko commented 5 months ago

Yes if you want to keep test on live API responses it can be done this way. We can also have 2 test files, one for live testing and the other for local and run them independently (but this only assuming that in future all test will be still kept in one-file format).

Choice is up to you.

alat-rights commented 5 months ago

Yup! Let's keep it all in simple_tests.py for now. We can always split into multiple files in the future.