Open Domejko opened 5 months ago
Hi!
Yes, tests are currently run live via the Anthropic API.
While I'm open to mocking, this is not currently considered a bug.
Can you get yourself an Anthropic API key here, and then add os.environ["ANTHROPIC_API_KEY"] = your_api_key
to your local copy of the test code?
Definitely want to automate this with GitHub Actions in the future.
I have already set up an account and used valid generated api_key
but then it was demanding to top up my credits, there is a option go get 5$ for free but I'm not convinced to give my phone number left and right.
I'll will try to mock that return value so that testing will be independent of valid api_key.
Ah, I see!
I think a graceful way to handle this would be to keep the existing functions as-is and add new ones with mocked outputs.
Proposal:
TestFunctionsLocal
and TestFunctionsAPI
?), and then take an argument from the terminal to decide whether to run both or to only run TestFunctionsLocal
.Yes if you want to keep test on live API responses it can be done this way. We can also have 2 test files, one for live testing and the other for local and run them independently (but this only assuming that in future all test will be still kept in one-file format).
Choice is up to you.
Yup! Let's keep it all in simple_tests.py
for now. We can always split into multiple files in the future.
While I wanted to perform tests I did encounter this error on 12 from 16 test cases:
Test could not be performed without Claude account with resources on it. I think that for general testing purposes we will need to mock Anthropic
client.messages.create
.