search

aiverify-foundation / moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
https://aiverify-foundation.github.io/moonshot/
Apache License 2.0
173 stars 34 forks source link

Operation failed when running a cookbook and the error refers to a model I'm not evaluating #283

Closed fcanogab closed 2 months ago

fcanogab commented 3 months ago

While running run_cookbook, I'm getting this error:

2024-08-01 17:09:02,512 [WARNING][connector.py::wrapper(49)] Operation failed. Error code: 404 - {'object': 'error', 'message': 'The model merlinite-7b-lab-Q4_K_M does not exist.', 'type': 'NotFoundError', 'param': None, 'code': 404} - Retrying..

The execution continues, it is not blocked. However, I don't know what this errors means. In addition, it refers to merlinite-7b-lab-Q4_K_M which is NOT the model I'm evaluating right now. It is in fact a model I have added in relation to another endpoint, but not in relation to the one I'm evaluating in this execution.

fcanogab commented 3 months ago

I fixed this by deleted existing runners. If I run a cookbook and while it is being executed I press Ctrl+C, if then I run another cookbook, are both executed? That might explain it, but I don't know if that's expected behaviour.

imda-kelvinkok commented 2 months ago

Hi @fcanogab. Thank you for raising this issue.

Sorry what do you mean by adding a model in relation to another endpoint? Can I know the endpoint you are using? Perhaps if it is possible can you pass us the configuration file so we can try to replicate this issue on our end?

Regarding the issue on the runners, for now there is no graceful shutdown. So doing a Ctrl+C may not shut down the current runner properly. As a temporary get around, if you want to stop the current run, can you exit the CLI and restart the CLI? We are currently working on a fix to properly stop a run. We will update again when this fix is live.

fcanogab commented 2 months ago

@imda-kelvinkok it sounds like the issue I'm hitting is related to the runners not being closed gracefully. If I'm able to reproduct the issue, I'll get back to you, but I think that this will be fixed when you commit the patch you mentioned.