Sequential Execution and Duplicate Validation in guardrails Library

[vishal-ushur]

Describe the bug When utilizing the Guard class from the guardrails library and employing multiple validators, the validators are executed sequentially instead of concurrently. Additionally, it appears that each validator is executed twice, leading to longer-than-expected execution times.

To Reproduce

1. Define multiple validators utilizing the @register_validator decorator.
2. Instantiate a Guard object and add the validators using the use_many method.
3. Invoke the validate method of the Guard object with a value that triggers both validators to fail.

Expected behavior The validators should execute concurrently, ideally leveraging parallelism, to reduce the overall execution time. Each validator should only be executed once per validation attempt.

Library version: Version 0.4.3

Additional context It's observed that the execution time increases significantly when multiple validators are employed, particularly when each validator includes a time-consuming operation like the time.sleep function. This behavior may significantly impact the performance of applications utilizing the guardrails library, especially in scenarios where real-time validation is crucial. Additionally, it's unclear why each validator appears to be executed twice per validation attempt, which warrants further investigation.

from guardrails.validators import Validator, register_validator, ValidationResult, PassResult, FailResult
from guardrails import Guard, OnFailAction
from typing import Dict, Any
import time

@register_validator(name="guardrails/contains-llm", data_type="string")
class ContainsLLM(Validator):

    check_word = "llm"

    def validate(self, value: Any, metadata: Dict) -> ValidationResult:

        # Push the move onto the board.
        print('I will sleep for 5 seconds.')
        time.sleep(5)
        if self.check_word in value:
            return PassResult()
        else:
            return FailResult(
                error_message=f"Value {value} is not a valid reply."
            )

@register_validator(name="guardrails/contains-hto", data_type="string")
class ContainsHTO(Validator):

    check_word = "hto"
    def validate(self, value: Any, metadata: Dict) -> ValidationResult:

        # Push the move onto the board.
        print('I will sleep for 2 seconds.')
        time.sleep(2)
        if self.check_word in value:
            return PassResult()
        else:
            return FailResult(
                error_message=f"Value {value} is not a valid reply."
            )

start_time = time.time()

guard = Guard().use_many(
    ContainsHTO(on_fail=OnFailAction.EXCEPTION),
    ContainsLLM(on_fail=OnFailAction.EXCEPTION),
)

try:
    guard.validate(
        """he gets"""
    )  # Both the guardrails fail
except Exception as e:
    print(e)
end_time = time.time()
print(f"Time taken: {end_time - start_time} seconds")

Code Output

I will sleep for 2 seconds.
I will sleep for 2 seconds.
I will sleep for 5 seconds.
I will sleep for 5 seconds.
Validation failed for field with errors: Value he gets is not a valid reply., Value he gets is not a valid reply.
Time taken: 14.024889945983887 seconds

^^ @zsimjee @ShreyaR @thekaranacharya @CalebCourier

[zsimjee]

Hi @vishal-ushur I'll look into this!

[CalebCourier]

@zsimjee just saw this when glancing through validator_service for other reasons. Looks to be related: https://github.com/guardrails-ai/guardrails/pull/756

[vishal-ushur]

@zsimjee @CalebCourier I'm curious, why does it appear that the validators weren't invoked concurrently here? I noticed in the codebase that AsyncValidatorService is designed for this specific purpose, and from examining the execution traces, I observed that the validate method of AsyncValidatorService is being called.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.