None of the tutorials work

[naskovai]

Hi, the framework and paper look very promising, but, unfortunately, I've been unable to get any of the tutorials to work. Neither in a Colab notebook nor locally on my mac.

When I try with a Colab notebook, I get the same error as https://github.com/stanford-futuredata/ARES/issues/46

When I try locally on my mac, I cannot even import ares

\>\>\> from ares import ARES

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/<placeholder>/Development/ARES/ares/__init__.py", line 1, in <module>
    from .ares import ARES
  File "/Users/<placeholder>/Development/ARES/ares/ares.py", line 3, in <module>
    from .rag_scoring import rag_scoring_config
  File "/Users/<placeholder>/Development/ARES/ares/rag_scoring.py", line 1, in <module>
    from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import begin
  File "/Users/<placeholder>/Development/ARES/ares/RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py", line 47, in <module>
    from ares.RAG_Automatic_Evaluation.Evaluation_Functions import (
  File "/Users/<>/Development/ARES/ares/RAG_Automatic_Evaluation/Evaluation_Functions.py", line 24, in <module>
    from vllm import LLM
ModuleNotFoundError: No module named 'vllm'

vllm requires linux - https://docs.vllm.ai/en/latest/getting_started/installation.html

Therefore, sadly the nice looking documentation is quite deceiving, as the ARES lib is utterly unusable at the moment.

[robbym-dev]

Hi @naskovai

The issue was resolved in the previous pip update, could you verify what ARES pip version you are currently using? Regarding the Colab notebook, we are currently reworking it to match the new pip version, it will get updated shortly.

[naskovai]

Hi @robbym-dev, thanks for responding.

The version I get from pip install ares-ai is ares-ai-0.6.1.

When I run the first example from the README in a Colab notebook with an A100 GPU, I get no results. Code:

from ares import ARES

ues_idp_config = {
    "in_domain_prompts_dataset": "nq_few_shot_prompt_for_judge_scoring.tsv",
    "unlabeled_evaluation_set": "nq_unlabeled_output.tsv", 
    "model_choice" : "gpt-3.5-turbo-0125"
} 

ares = ARES(ues_idp=ues_idp_config)
results = ares.ues_idp()
print(results)
# {'Context Relevance Scores': [Score], 'Answer Faithfulness Scores': [Score], 'Answer Relevance Scores': [Score]}

Output:

Number of times did not extract Yes or No: 0
{'Context Relevance Scores': nan, 'Answer Faithfulness Scores': nan, 'Answer Relevance Scores': nan}

[robbym-dev]

Hi @naskovai,

Thank you for your response. The Colab notebook is currently being reworked, and the issues you're experiencing with no results will be resolved soon.

Were you able to resolve the vLLM import bug after updating to the new pip version?

[naskovai]

In the Colab environment I installed vLLM, so there is no such issue there. However, on a MacBook this is what happens when I try to import ares.

Successfully installed ares-ai-0.6.1
(myenv) ➜  ARES git:(main) ✗ python
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ares
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/naskovai/Development/ARES/ares/__init__.py", line 1, in <module>
    from .ares import ARES
  File "/Users/naskovai/Development/ARES/ares/ares.py", line 3, in <module>
    from .rag_scoring import rag_scoring_config
  File "/Users/naskovai/Development/ARES/ares/rag_scoring.py", line 1, in <module>
    from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import begin
  File "/Users/naskovai/Development/ARES/ares/RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py", line 47, in <module>
    from ares.RAG_Automatic_Evaluation.Evaluation_Functions import (
  File "/Users/naskovai/Development/ARES/ares/RAG_Automatic_Evaluation/Evaluation_Functions.py", line 24, in <module>
    from vllm import LLM
ModuleNotFoundError: No module named 'vllm'

[naskovai]

@robbym-dev curious, when do you think the Colab notebook will be updated?

[elsatch]

Hi @naskovai I thought vllm was available in Mac too, but it seems it is based on Triton and Triton is not supported on Mac. So there is no support to run vllm on Mac, as per https://github.com/vllm-project/vllm/issues/1441

Some people mention than compiling Triton from source would make it work: https://github.com/triton-lang/triton/issues/3443 (I am not so sure about the imports later on but... )

[robbym-dev]

Hi @naskovai,

The Colab notebook is expected to be updated within the next couple of days.

The error you are facing when importing vLLM on your Mac is due to it not being installed. ARES is designed to make vLLM an optional dependency, as @elsatch pointed out, vLLM isn't available on Mac.

Unfortunately, we don't know of a way to install vLLM on Mac at this time, but the Triton compiling mentioned by @elsatch might be worth exploring.

Are you currently facing any other issues with the ares pip we can assist you with? Moving forward, we will address this in the documentation to avoid any further confusion.

[naskovai]

Thank you for all the support!

Looking forward to the updated Colab notebooks!

[robbym-dev]

Hi @naskovai,

The Google Colab notebook has been updated here. You should now be able to run the examples successfully. Please let us know if you encounter any further issues or have any questions.

Thank you for your patience!