NVIDIA / NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Other
4.16k stars 392 forks source link

self_check_output / output flows don't activate #270

Open dannielvieira-hotmart opened 9 months ago

dannielvieira-hotmart commented 9 months ago

I'm running the getting_started notebooks to experiment and understand how each Rails work, but none of the output rails declared work. For example, on the [https://github.com/NVIDIA/NeMo-Guardrails/tree/develop/docs/getting_started/5_output_rails](Output Rail) introuction notebook, when running the section that uses the output flow declared in the config.yml, it doesn't get activated when calling the rails.generate. Why is that?

Prints of the llm_calls_summary result, config.yml and prompts.yml. I followed along the tutorial and got to this, does anyone what happened?

image image image
drazvan commented 9 months ago

Hi @dannielvieira-hotmart! It seems like it's somehow still using the config from the previous step, where the output rail was not configured. Can you make sure all the steps from the notebook run in order. Also, for a sanity check, do also add the following before the print statement:

print(rails.config.rails)
print(info.llm_calls[2].prompt)

You should see something similar to

config=RailsConfigData(fact_checking=FactCheckingRailConfig(parameters={}, fallback_to_self_check=False), sensitive_data_detection=SensitiveDataDetection(recognizers=[], input=SensitiveDataDetectionOptions(entities=[], mask_token='*'), output=SensitiveDataDetectionOptions(entities=[], mask_token='*'), retrieval=SensitiveDataDetectionOptions(entities=[], mask_token='*'))) input=InputRails(flows=['self check input']) output=OutputRails(flows=['self check output', 'check blocked terms']) retrieval=RetrievalRails(flows=[]) dialog=DialogRails(single_call=SingleCallConfig(enabled=False, fallback_to_multiple_calls=True), user_messages=UserMessagesConfig(embeddings_only=False))
dannielvieira-hotmart commented 9 months ago

Hello @drazvan. I restarted the kernel and ran in order, but still somehow is not loading current folder config, in it. Here is the output of the rails.config.rails:

image

I tested same notebook, but running on a different kernel and folder, and it ran just fine. Really can't tell why is using previous step.

drazvan commented 9 months ago

Thanks for the follow up @dannielvieira-hotmart. I guess there could be some weird bug with the way the Jupyter notebook interacts with the file system?? hard to tell.