Closed aaravnavani closed 2 months ago
We should add the same fixes for local inference to the validator's main.py
.
@CalebCourier from the tests that I ran, it seems to detect apple
as a competitor with local inference. This script for example:
# Import Guard and Validator
from guardrails import Guard
from guardrails.hub import CompetitorCheck
# Setup Guard
guard = Guard().use(
CompetitorCheck, ["Apple", "Samsung"], "exception", use_local=True,
)
response = guard.validate("apple just released a new iPhone.") # Validator fails
print(response)
prints guardrails.errors.ValidationError: Validation failed for field with errors: Found the following competitors: [['Apple']]. Please avoid naming those competitors next time
It seems to be handled in this line.
@aaravnavani that's only the initial filter. It also needs to be handled here and potentially in filtering methods.
As a side note, the fact that it's listing competitors as an Array<Array<String>>
seems off.
@CalebCourier from the tests that I ran, it seems to detect
apple
as a competitor with local inference. This script for example:# Import Guard and Validator from guardrails import Guard from guardrails.hub import CompetitorCheck # Setup Guard guard = Guard().use( CompetitorCheck, ["Apple", "Samsung"], "exception", use_local=True, ) response = guard.validate("apple just released a new iPhone.") # Validator fails print(response)
prints
guardrails.errors.ValidationError: Validation failed for field with errors: Found the following competitors: [['Apple']]. Please avoid naming those competitors next time
It seems to be handled in this line.
I'm not seeing that behaviour locally. If I try to run that exact code snippet this is what I get:
ValidationOutcome(
call_id='14844740304',
raw_llm_output='apple just released a new iPhone.',
validated_output='apple just released a new iPhone.',
reask=None,
validation_passed=True,
error=None
)
This PR fixes this issue.
For example, if the competitor that is passed in is "Apple" and we are validating the text
Fun fact about apple is that apple is both a fruit and company. apples come in many different colors and flavors. What if apple is a tech company that makes phones, computers, and tablets? apple is headquartered in Cupertino, California
, it properly identifies lowercaseapple
as a competitor.Curl command to test with fastapi setup running on
localhost
:gives output:
{"modelname":"en_core_web_trf","modelversion":"1","outputs":[{"name":"result0","shape":[1],"data":[["apple","apple","apple"]],"datatype":"BYTES"}]}