Problem with Qwen

[rdaigle007]

This is so cool. It works nicely with the default llama model in the code.

Note it doesn't work when trying to use the following model: 'Qwen/Qwen1.5-110B-Chat-demo'

Exception as follows:

  File "/Volumes/ExtremePro/safety-class/gradio-client-any-llm/CustomLLM/CustomLlm.py", line 55, in _call
    result = chatbot.predict(   #.submit for streaming effect / .predict for normal output
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/ExtremePro/virtual-envs/env-safety-class/lib/python3.11/site-packages/gradio_client/client.py", line 455, in predict
    inferred_fn_index = self._infer_fn_index(api_name, fn_index)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/ExtremePro/virtual-envs/env-safety-class/lib/python3.11/site-packages/gradio_client/client.py", line 781, in _infer_fn_index
    raise ValueError(error_message)
ValueError: Cannot find a function with `api_name`: /chat.
(env-safety-class)Russells-MacBook-Pro-2:CustomLLM$

Not sure if problem is won't work for models otrher than llama3... is if it has to do with the gradio app in the HuggingFace Space.

Would be very cool if there was a way to generalize this CustomLLM class you created to work for more models/HF-spaces.

[rdaigle007]

oh! I learned from your other document at https://generativeai.pub/i-hacked-the-ai-agents-now-you-can-have-them-all-for-free-08cae7d29714

It shows how to check the gradio app in the HuggingFace Spaces to see if it has API link at bottom of app, click it to find the API.

So there is no really way to make your CustomLLM class generic to try to handle different scenarios, and give the user feedback?

This was very interesting example to play with.

[fabiomatricardi]

Thanks for the feedback. the explanation for other gradio_API is in my Medium article.

It cannot be generalized, because the EndPoints are not standard OpenAI compatible APIs.

this is the details of Qwen110b: you can see that the output is not a string, but a touple of dicts So for Qwen you should do something like this:

from gradio_client import Client

yourHFtoken = "hf_xxxxxxxxxxxxxxxxx" #here your HF token
client = Client("Qwen/Qwen1.5-110B-Chat-demo", hf_token=yourHFtoken)
result = client.predict(
                query="Why you should learn AI?",
                history=[],
                system="You are a helpful assistant.",
                api_name="/model_chat"
)
print(result)

this will print you the kind of expected output,

Loaded as API: https://qwen-qwen1-5-110b-chat-demo.hf.space ✔
('', [['Why you should learn AI?', "Learning about Artificial Intelligence (AI) can be valuable for several reasons, applicable to a wide range of individuals and industries. Here are some key reasons why:\n\n1. **Career Opportunities**: AI is a rapidly growing field with high demand for skilled professionals. Jobs in AI, such as data analysts, machine learning engineers, AI researchers, and AI ethicists, offer lucrative career paths.\n\n2. **Innovation and Problem Solving**: AI technologies are transforming various sectors, from healthcare to finance, manufacturing, and more. Understanding AI allows you to contribute to innovative solutions to complex problems.\n\n3. **Improved Decision Making**: AI can process vast amounts of data and provide insights that humans might miss. Learning AI can help you make better decisions based on data-driven analysis.\n\n4. **Automation and Efficiency**: AI is driving automation, which can lead to increased efficiency and productivity in many industries. Understanding AI helps you adapt to changes in the workplace and leverage automation for better outcomes.\n\n5. **Personalization and User Experience**: AI powers personalized experiences, from recommendations on streaming platforms to tailored shopping experiences. Knowing AI can help you create products or services that better meet users' needs.\n\n6. **Ethical Considerations**: As AI becomes more prevalent, understanding its ethical implications is crucial. It helps in designing responsible AI systems that align with societal values and prevent biases.\n\n7. **Future-Proofing Skills**: AI is not just a trend; it's shaping the future. By learning AI, you position yourself to be more adaptable to future technological advancements.\n\n8. **Global Competitiveness**: In a global economy increasingly driven by technology, AI literacy gives individuals and countries a competitive edge.\n\n9. **Scientific Research**: AI is also enhancing research across disciplines, from drug discovery to climate science, by enabling faster analysis and pattern recognition.\n\n10. **Understanding the World Around Us**: AI influences how we interact with technology daily. Being informed about AI can help you make better choices as a consumer and citizen, regarding privacy, security, and the impact of AI on society.\n\nIn summary, learning AI is a strategic investment in your personal and professional development, equipping you with skills necessary for navigating our rapidly evolving digital world."]], 'You are a helpful assistant.')

so your iterator is encapsulated in

result[1][0][1]

from there you iterate