search

guardrails-ai / guardrails

Adding guardrails to large language models.
https://www.guardrailsai.com/docs
Apache License 2.0
3.81k stars 286 forks source link

[feat] Support for GPU usage #907

Open AbhiPawar5 opened 1 month ago

AbhiPawar5 commented 1 month ago

Description For Profanity and NSFW detection, the code uses a transformer pipeline under the hood. However, the device argument should be passed to use the best hardware accelerator.

Why is this needed This is a general feature to speedup the pipeline inference significantly

Implementation details Adding device with proper integer should solve the problem. More details here - https://huggingface.co/transformers/v3.0.2/main_classes/pipelines.html#transformers.Pipeline

End result Across multiple pipelines and use cases.

wylansford commented 1 month ago

Good catch!

I've added GPU support for nsfw text here: https://github.com/guardrails-ai/nsfw_text

For the profanity check, did you mean https://github.com/guardrails-ai/profanity_free ?

I believe this validator runs off of sci-kit on the backend, so I am not sure if it provides GPU support natively.

AbhiPawar5 commented 1 month ago

Thanks! I see the log saying pipeline is loaded and device set as CPU. I believe its still a Hugging Face pipeline

On Thu, 11 Jul, 2024, 10:09 pm Wyatt Lansford, @.***> wrote:

Good catch!

I've added GPU support for nsfw text here: https://github.com/guardrails-ai/nsfw_text

For the profanity check, did you mean https://github.com/guardrails-ai/profanity_free ?

I believe this validator runs off of sci-kit on the backend, so I am not sure if it provides GPU support natively.

— Reply to this email directly, view it on GitHub https://github.com/guardrails-ai/guardrails/issues/907#issuecomment-2223403445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK3AUDEMIXLLG23RHOPLUSLZL2YLPAVCNFSM6AAAAABKWGDUJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRTGQYDGNBUGU . You are receiving this because you authored the thread.Message ID: @.***>

wylansford commented 1 month ago

Can you provide the logs? From what I can tell it uses,

https://github.com/vzhou842/profanity-check

Which is just an SVM on the backend. There shouldn't be any huggingface calls made for it

here is the code: https://github.com/guardrails-ai/profanity_free/blob/main/validator/main.py