RuntimeError: Bad response to Guidance request (phi-3.5-mini-instruct serverless api)

[Remilistrasza]

The bug Received the following error when using gen() with phi-3.5-mini

RuntimeError: Bad response to Guidance request
Request: https://model_name.region.models.ai.azure.com/guidance (authorization: Bearer XX...XX)
Response: 404 Not Found
NOT FOUND

Traceback

```Error --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Cell In[2], line 4 2 phi3_lm += "What is the capital of Australia?" 3 with assistant(): ----> 4 phi3_lm += "The capital of Australia is " + gen("answer", stop="\n", temperature=0) File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_model.py:486, in Model.__add__(self, value) 484 # run stateless functions (grammar nodes) 485 elif isinstance(value, GrammarFunction): --> 486 out = lm._run_stateless(value) 488 # run stateful functions 489 else: 490 out = value(lm) File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_model.py:692, in Model._run_stateless(self, stateless_function, temperature, top_p, n) 690 delayed_bytes = b"" 691 # last_is_generated = False --> 692 for chunk in gen_obj: 693 694 # we make everything full probability if we are not computing uncertainty 695 # if not self.engine.compute_log_probs: 696 # chunk.new_bytes_prob = 1.0 697 698 # convert the bytes to a string (delaying if we don't yet have a valid unicode string) 699 lm.token_count += chunk.new_token_count 700 chunk.new_bytes = delayed_bytes + chunk.new_bytes File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_azure_guidance.py:73, in AzureGuidanceEngine.__call__(self, parser, grammar, ensure_bos_token) 71 except: 72 pass ---> 73 raise RuntimeError( 74 f"Bad response to Guidance request\nRequest: {info}\n" 75 + f"Response: {resp.status_code} {resp.reason}\n{text}" 76 ) 78 for line in resp.iter_lines(): 79 if not line: RuntimeError: Bad response to Guidance request Request: https://model_name.region.models.ai.azure.com/guidance (authorization: Bearer XX...XX) Response: 404 Not Found NOT FOUND ```

It was working back in Oct, but after 3 weeks I cannot re-run the same code I had before. Found a discussion of the similar topic: https://github.com/guidance-ai/guidance/discussions/1041#discussioncomment-11141599, where other people are having the same issue, but I didn't find any working solution.

To Reproduce

1. Create a serverless phi-3.5-mini-instruct endpoint following this guide: https://github.com/microsoft/Phi-3CookBook/blob/main/code/01.Introduce/guidance.ipynb
2. Run the following code:

   
   from guidance import gen, user, assistant
   from guidance.models import AzureGuidance

phi3_url = "https://model-name.region.models.ai.azure.com" phi3_api_key = "api_key" phi3_lm = AzureGuidance(f"{phi3_url}/guidance#auth={phi3_api_key}")

with user(): phi3_lm += "What is the capital of Australia?" with assistant(): phi3_lm += "The capital of Australia is " + gen("answer", stop="\n", temperature=0)

I also tried to add /chat/completions to the route, but got the same error

**System info (please complete the following information):**
 - OS: Windows 11
 - Guidance Version: 0.2.0rc1

[jmandivarapu]

I am facing similar issue. Create issued on microsoft phi3 cookbook too. Still no resolution (https://github.com/microsoft/Phi-3CookBook/issues/222)

[Harsha-Nori]

Hi @Remilistrasza and @jmandivarapu --

So sorry about that! We had to temporarily take down the experimental Phi-3.5 integration, but we will be coming back + on more Azure AI models in the very near future. I will update this thread when we are live again on Azure AI. It's early days for us on testing how this experimental integration will go into Azure AI, so really appreciate your patience with us here!

-Harsha

[jmandivarapu]

Thank you for your response @Harsha-Nori. Can I know if there is any timeline for this? So it will give us an idea of waiting or looking for other solutions.