Sherpa does not accurately extract numeric values from search results

[YujingYang666777]

Description: when searching for the annual salary for several jobs, the results given by the LLM are exactly the same. Probably the searched result can be more accurate regarding the specific question.

For example: User: @Sherpa What is the annual salary for an entry level software engineer in Canada? Sherpa: The average annual salary for an entry-level software engineer in Canada ranges from $68,207 to $110,963. Source: Levels.fyi

However, when clicking into the URL, the number display on the website is 5.07 - 11.13 instead of the number claimed by Sherpa.

[amirfz]

@YujingYang666777 could you add some examples to the description?

[amirfz]

@YujingYang666777 talk to @johnnykfeng because he had to solve this issue in a different project

[oshoma]

@YujingYang666777 @johnnykfeng is this issue still occurring? Any comments to add?

[oshoma]

I just tried this again. It's still happening.

Osh: https://github.com/sherpa What is the annual salary for an entry level software engineer in Canada? Sherpa: The annual salary for an entry level software engineer in Canada can vary. According to Glassdoor, the median pay for entry level software engineers in Canada is CA$104,203 [1]. Another source, Levels.fyi, states that the average salary range for entry level software engineers in Canada is from CA$67,576 to CA$107,680 [2]. It's important to note that salaries can also depend on factors such as location and experience.

The $104,203 number does not appear anywhere on the 1st URL page. For the 2nd URL, the actual range on the web page is $84K - $130K/yr.

[amirfz]

part of the evidence verification that we looked at is dealing with numbers and entities specifically. maybe once we have a baseline "evidence verification" and "citation" then we can break this down more specifically to sub tasks that need to be handled?

On Tue, Dec 5, 2023 at 11:26 AM Oshoma Momoh @.***> wrote:

I just tried this again. It's still happening.

Osh: https://github.com/sherpa What is the annual salary for an entry level software engineer in Canada? Sherpa: The annual salary for an entry level software engineer in Canada can vary. According to Glassdoor, the median pay for entry level software engineers in Canada is CA$104,203 [1]. Another source, Levels.fyi, states that the average salary range for entry level software engineers in Canada is from CA$67,576 to CA$107,680 [2]. It's important to note that salaries can also depend on factors such as location and experience.

The $104,203 number does not appear anywhere on the 1st URL page. For the 2nd URL, the actual range on the web page is $84K - $130K/yr.

— Reply to this email directly, view it on GitHub https://github.com/Aggregate-Intellect/sherpa/issues/80#issuecomment-1841142861, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4OK5ILECVAQTLAPKOP2NLYH5DLJAVCNFSM6AAAAAA3LP35GCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRGE2DEOBWGE . You are receiving this because you commented.Message ID: @.***>

[amirfz]

ok, to add more concrete details. this is also related and can probably borrow implementation ideas from #241

• create a critic agent for the QA agent @20001LastOrder
• extract numbers from the response (regex?) @Eyobyb
• check if those numbers exist exactly as they are in the references then accept the answer and send it to the user @Eyobyb
• if not, then respond back to the QA agent with something like "the numbers presented in your answer do not appear in the references. please try again responding and including the correct numbers"

ps. this can be also done for entities that are important. pps. this doesn't solve the problem of a number existing but not being what the user actually wanted, but this would be a good baseline for obvious errors

[oshoma]

@Eyobyb is working on #260 in relation to this.

[20001LastOrder]

Closed by #277