[BUG] Setting wrong HuggingFace Hub models prevent the Cat from starting

[nicola-corbellini]

Describe the bug The HuggingFace Hub setting only accepts few categories of NLP models. Setting a LLM from a not valid category saves the setting in the db, but then prevents the Cat from starting. Hence, changing back to another valid model is not possible anymore. The only solution is to delete metadata-v3.db.

To Reproduce Steps to reproduce the behavior (as example):

1. Go to the Setting page
2. Configure LLM with HuggingFace Hub
3. Set the repo id from any model from HuggingFace with a tag different from 'text2text-generation', 'text-generation' or 'summarization' (e.g. use a 'Conversational' model like this: microsoft/DialoGPT-medium)
4. See error:

   cheshire_cat_core           |   Got invalid task conversational, currently only ('text2text-generation', 'text-generation', 'summarization') are supported (type=value_error)

5. Change back to any other valid LLM (e.g. ChatGPT)
6. See error:

   pydantic.error_wrappers.ValidationError: 1 validation error for HuggingFaceHub
   cheshire_cat_core           | __root__
   cheshire_cat_core           |   Got invalid task conversational, currently only ('text2text-generation', 'text-generation', 'summarization') are supported (type=value_error)
   cheshire_cat_core           | 
   cheshire_cat_core           | ERROR:    Application startup failed. Exiting.

Expected behavior The Cat should check if the model is correct before getting the configuration and prevent setting it to no crash. Additionally, the admin could notify the event.

Additional context Specifically, the app crashes here

[pieroit]

A first fix could be to use LLMDefaultConfig when there is an exception during model instantiation. So if anything goes wrong, the cat does not die (he is already 6 lives out of 7)

As a second step we can specialize the exception to dealt with specific LLMs

[SantoSimone]

Hi, I was getting my hands dirty with this repository by looking at this issue. A naive approach is to wrap the code into a try - except block, like this:

        try:
            llm = FactoryClass.get_llm_from_config(selected_llm_config.value)
        except Exception as e:
            import traceback
            traceback.print_exc()
            llm = llms.LLMDefaultConfig.get_llm_from_config({})

But now I get some error due to qdrant.. cheshire_cat_core | qdrant_client.http.exceptions.ResponseHandlingException: timed out

I don't think this is related to the change I made, but maybe I am wrong, is this something you already faced?

[nicola-corbellini]

@SantoSimone It didn't happen to me, could you share the full error trace please? Is this happening right after you set the LLM?

My best guess is that this error is due to the tool embedding that happens when bootstrapping the Cat.

[SantoSimone]

I had to reboot the pc in the meanwhile and it seems I do not get that error now, strange..

Anyway, is the naive solution what we were looking for? Should I make a PR?

[nicola-corbellini]

I think it is, let's hear from @pieroit

[pieroit]

@SantoSimone @nicola-corbellini yes let's go with the naive solution, and if demand for HF support raises we come back to improve ;)