search

langchain-ai / langchain-google

MIT License
74 stars 78 forks source link

Support safety_settings `method` parameter #243

Closed kardiff18 closed 2 weeks ago

kardiff18 commented 1 month ago

With Vertex, there is the case that false positives occasionally arise in the safety settings due to using the "probability" method as default. To help fix the false positives, it would be desirable to switch to "severity" instead. This is not supposed in the way that LangChain accepts the safety_settings.

To fix the safety settings, it would likely break the current implementation since it is a dictionary rather than a proper object with the parameter names from Vertex. However, I think this is important to change to. If there are alternatives, please let me know. More details below and documentation from Vertex here.

Since langchain safety filters are in the format {HarmCategory.HARM_CATEGORY_UNSPECIFIED: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE, it doesn't seem very conducive to doing the same in the Vertex SDK, but in Vertex I can do this and use severity rather than probability (the default) which is extremely helpful in reducing false positives

SafetySetting(
    category=SafetySetting.HarmCategory.HARM_CATEGORY_HARASSMENT,
    threshold=SafetySetting.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,
    method: SafetySetting.HarmBlockMethod.SEVERITY
   )
yeesian commented 1 month ago

I think this might be doable if the validation for SafetySettings is fixed in https://github.com/langchain-ai/langchain-google/blob/eedb0d2e105c3573388ab2470702871efcddaab2/libs/vertexai/langchain_google_vertexai/_base.py#L142C32-L142C50

Currently setting it in the non-dict format will fail with the following error message:

ValidationError: 1 validation error for ChatVertexAI
safety_settings
  value is not a valid dict (type=type_error.dict)
yeesian commented 1 month ago

Investigating further, it looks like this has been fixed in https://github.com/langchain-ai/langchain-google/commit/4435b06ddd53cb06eddbd11f987b4b37bcf91d0b and we just need a new release

lkuligin commented 2 weeks ago

it looks like this one can be closed

kardiff18 commented 1 week ago

I need to debug this further, but FYI it doesn't appear that this is being set when it's sent to Vertex (the method is not in the request object).

kardiff18 commented 1 week ago

NVM - we are setting it, we just don't see it returned when using debug. I think when langchain gets the safetysetting back we need to return the severity parameters as well as the probability parameters.