[Question]: How to use Llama Pack - Resume Screener without OpenAI API KEY ?

[ByUnal]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I want to use Llama Pack - Resume Screener with the embedding or LLM model which I defined. Even tough I followed the documentation precisely. However, Resume Screener still says try to use embed_model = 'local'. How can I fix this ?

Here is the code I used for local embedding:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings

# loads BAAI/bge-small-en-v1.5
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

Settings.embed_model = embed_model

The code I tried to run as shown here:

resume_screener = ResumeScreenerPack(
    job_description=meta_jd,
    criteria=[
        "2+ years of experience in one or more of the following areas: machine learning, recommendation systems, pattern recognition, data mining, artificial intelligence, or related technical field",
        "Experience demonstrating technical leadership working with teams, owning projects, defining and setting technical direction for projects",
        "Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.",
    ],
)

The error I get:

ValueError: 
******
Could not load OpenAI embedding model. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

Consider using embed_model='local'.
Visit our documentation for more embedding options: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules

[dosubot[bot]]

It's great to see that you're trying to use Llama Pack - Resume Screener with a custom embedding model! It seems like you're encountering an error related to the OpenAI API key. If you'd like to contribute a fix for this issue, you can create a pull request with your proposed solution. Your contribution would be valuable to the community. Keep up the good work!

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[ravi03071991]

@ByUnal You need to pass the llm (other than openai in your case here). You can check the source code to get better understanding of it. Something like the following code snippet should ideally work.

resume_screener = ResumeScreenerPack(
    llm=...
    job_description=meta_jd,
    criteria=[
        "2+ years of experience in one or more of the following areas: machine learning, recommendation systems, pattern recognition, data mining, artificial intelligence, or related technical field",
        "Experience demonstrating technical leadership working with teams, owning projects, defining and setting technical direction for projects",
        "Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.",
    ],
)

[anoopshrma]

Yeah, download the pack, replace the llm with your own llm object. You are good to go!

[ByUnal]

@ravi03071991 Yeah, I've read the document and tried what you just said, but it also doesn't work.

from llama_index.core import PromptTemplate

# Transform a string into input zephyr-specific input
def completion_to_prompt(completion):
    return f"<|system|>\n</s>\n<|user|>\n{completion}</s>\n<|assistant|>\n"

# Transform a list of chat messages into zephyr-specific input
def messages_to_prompt(messages):
    prompt = ""
    for message in messages:
        if message.role == "system":
            prompt += f"<|system|>\n{message.content}</s>\n"
        elif message.role == "user":
            prompt += f"<|user|>\n{message.content}</s>\n"
        elif message.role == "assistant":
            prompt += f"<|assistant|>\n{message.content}</s>\n"

    # ensure we start with a system prompt, insert blank if needed
    if not prompt.startswith("<|system|>\n"):
        prompt = "<|system|>\n</s>\n" + prompt

    # add final assistant prompt
    prompt = prompt + "<|assistant|>\n"

    return prompt

import torch
from llama_index.llms.huggingface import HuggingFaceLLM
from llama_index.core import Settings

llm = HuggingFaceLLM(
    model_name="microsoft/phi-2",
    tokenizer_name="microsoft/phi-2",
    context_window=3900,
    max_new_tokens=256,
    generate_kwargs={"temperature": 0.4, "top_k": 50, "top_p": 0.95},
    messages_to_prompt=messages_to_prompt,
    completion_to_prompt=completion_to_prompt,
    device_map="auto")

I've created following custom LLM by using the code above, then I tried:

resume_screener = ResumeScreenerPack(
    llm = llm,
    job_description=meta_jd,
    criteria=[
        "2+ years of experience in one or more of the following areas: machine learning, recommendation systems, pattern recognition, data mining, artificial intelligence, or related technical field",
        "Experience demonstrating technical leadership working with teams, owning projects, defining and setting technical direction for projects",
        "Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.",
    ],
)

But it gives error as such:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/llama_index/core/embeddings/utils.py](https://localhost:8080/#) in resolve_embed_model(embed_model, callback_manager)
     58             embed_model = OpenAIEmbedding()
---> 59             validate_openai_api_key(embed_model.api_key)
     60         except ImportError:

5 frames
ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/llama_index/core/embeddings/utils.py](https://localhost:8080/#) in resolve_embed_model(embed_model, callback_manager)
     64             )
     65         except ValueError as e:
---> 66             raise ValueError(
     67                 "\n******\n"
     68                 "Could not load OpenAI embedding model. "

ValueError: 
******
Could not load OpenAI embedding model. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

Consider using embed_model='local'.
Visit our documentation for more embedding options: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules
******

[anoopshrma]

If you see the error, it is coming in the embedding side.

Do this:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

Also add the llm to Settings as well. Settings.llm=llm

[ByUnal]

@anoopshrma Still Same. I've already tried those. Here are the outputs and code.

Code:

from llama_index.core import PromptTemplate
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

# Transform a string into input zephyr-specific input
def completion_to_prompt(completion):
    return f"<|system|>\n</s>\n<|user|>\n{completion}</s>\n<|assistant|>\n"

# Transform a list of chat messages into zephyr-specific input
def messages_to_prompt(messages):
    prompt = ""
    for message in messages:
        if message.role == "system":
            prompt += f"<|system|>\n{message.content}</s>\n"
        elif message.role == "user":
            prompt += f"<|user|>\n{message.content}</s>\n"
        elif message.role == "assistant":
            prompt += f"<|assistant|>\n{message.content}</s>\n"

    # ensure we start with a system prompt, insert blank if needed
    if not prompt.startswith("<|system|>\n"):
        prompt = "<|system|>\n</s>\n" + prompt

    # add final assistant prompt
    prompt = prompt + "<|assistant|>\n"

    return prompt

import torch
from llama_index.llms.huggingface import HuggingFaceLLM
from llama_index.core import Settings

Settings.llm = HuggingFaceLLM(
    model_name="microsoft/phi-2",
    tokenizer_name="microsoft/phi-2",
    context_window=3900,
    max_new_tokens=256,
    generate_kwargs={"temperature": 0.4, "top_k": 50, "top_p": 0.95},
    messages_to_prompt=messages_to_prompt,
    completion_to_prompt=completion_to_prompt,
    device_map="auto"
    )

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

resume_screener = ResumeScreenerPack(
    llm = Settings.llm,
    job_description=meta_jd,
    criteria=[
        "2+ years of experience in one or more of the following areas: machine learning, recommendation systems, pattern recognition, data mining, artificial intelligence, or related technical field",
        "Experience demonstrating technical leadership working with teams, owning projects, defining and setting technical direction for projects",
        "Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.",
    ]
)

And the error still same above.

[anoopshrma]

Can you share the Latest error

[ByUnal]

@anoopshrma I've already told you, same with the error above I shared. Here is the image of error:

Maybe you can retry my code to reproduce error I got.

[anoopshrma]

I think you'll have to add the embedding model here as well! https://github.com/run-llama/llama_index/blob/main/llama-index-packs/llama-index-packs-resume-screener/llama_index/packs/resume_screener/base.py#L64

Since you already have the llamapack downloaded just make this change on service context line.

        service_context = ServiceContext.from_defaults(llm=llm,embed_model=Settings.embed_model)

[ByUnal]

Nothing change. I'm trying on this google colab by the way. So, please try to reproduce the error I encountered. If you can solve, please let me know.

[anoopshrma]

Nothing change. I'm trying on this google colab by the way. So, please try to reproduce the error I encountered. If you can solve, please let me know.

I tried passing the embed_model in the service_context and it worked. Try the below code

from pathlib import Path
 from typing import Any, Dict, List, Optional

 from llama_index.core import ServiceContext
 from llama_index.core.llama_pack.base import BaseLlamaPack
 from llama_index.core.response_synthesizers import TreeSummarize
 from llama_index.core.schema import NodeWithScore
 from llama_index.llms.openai import OpenAI
 from llama_index.readers.file import PDFReader
 from pydantic import BaseModel, Field

 # backwards compatibility
 try:
     from llama_index.core.llms.llm import LLM
 except ImportError:
     from llama_index.core.llms.base import LLM

 QUERY_TEMPLATE = """
 You are an expert resume reviewer.
 You job is to decide if the candidate pass the resume screen given the job description and a list of criteria:

 ### Job Description
 {job_description}

 ### Screening Criteria
 {criteria_str}
 """

 class CriteriaDecision(BaseModel):
     """The decision made based on a single criteria."""

     decision: bool = Field(description="The decision made based on the criteria")
     reasoning: str = Field(description="The reasoning behind the decision")

 class ResumeScreenerDecision(BaseModel):
     """The decision made by the resume screener."""

     criteria_decisions: List[CriteriaDecision] = Field(
         description="The decisions made based on the criteria"
     )
     overall_reasoning: str = Field(
         description="The reasoning behind the overall decision"
     )
     overall_decision: bool = Field(
         description="The overall decision made based on the criteria"
     )

 def _format_criteria_str(criteria: List[str]) -> str:
     criteria_str = ""
     for criterion in criteria:
         criteria_str += f"- {criterion}\n"
     return criteria_str

 class ResumeScreenerPack(BaseLlamaPack):
     def __init__(
         self, job_description: str, criteria: List[str], llm: Optional[LLM] = None
     ) -> None:
         self.reader = PDFReader()
         llm = llm or OpenAI(model="gpt-4")
         service_context = ServiceContext.from_defaults(llm=llm, embed_model=Settings.embed_model)
         self.synthesizer = TreeSummarize(
             output_cls=ResumeScreenerDecision, service_context=service_context
         )
         criteria_str = _format_criteria_str(criteria)
         self.query = QUERY_TEMPLATE.format(
             job_description=job_description, criteria_str=criteria_str
         )

     def get_modules(self) -> Dict[str, Any]:
         """Get modules."""
         return {"reader": self.reader, "synthesizer": self.synthesizer}

     def run(self, resume_path: str, *args: Any, **kwargs: Any) -> Any:
         """Run pack."""
         docs = self.reader.load_data(Path(resume_path))
         output = self.synthesizer.synthesize(
             query=self.query,
             nodes=[NodeWithScore(node=doc, score=1.0) for doc in docs],
         )
         return output.response

Then ccreate the resume screener object

resume_screener = ResumeScreenerDecision(llm=llm....)

make sure to define llm and embed model at the top of this

[ByUnal]

@anoopshrma Now, object has been created. I think it works from now on. Runtime has exploded due to over-usage of available of RAM, but it is fine now. Thanks for your time and effort.