-
Evaluate safety issue of LLMs from various aspects
(To be continued, open by JingFeng, Hongye, ...)
-
DeepEval docs have a Google Vertex AI implementation sample here:
https://docs.confident-ai.com/docs/metrics-introduction#google-vertexai-example
However, it has a number of issues:
1. It doesn…
-
I noticed in the paper that 'In total, we collected 2,000 malicious documents for training with an average number of tokens of 702.79'. However, I do not find these malicious documents within the repo…
-
Review the thread safety of the SDK to make sure there's no potential concurrency issues, particularly around state maintained in the global API object, clients, and evaluation context objects.
See…
-
**Describe the bug**
A clear and concise description of the bug.
**How To Reproduce the bug**
Steps to reproduce the behavior, how frequent can you experience the bug:
I am trying to hook up the…
-
# Task Name
Vehicle sounds classification
## Task Objective
The primary goal of this task is to evaluate the audio language model's capability to accurately recognize and classify different …
-
# Industrial Machine situation
This dataset is a sound dataset for malfunctioning industrial machine investigation and inspection
## Task Objective
In this task, we consider false positives and …
-
Sorry for creating another issue,
I am quite lost on how to submit the results of the attack for Claude-3 and GPT-4-0613.
Another question I have concerns the use of a benign dataset. I observed…
-
Hi
Could you share scripts that may reproduce the results in the paper ? Thanks.
I tried the generation and evaluation for safety using the following script on an Nvidia GPU. The results are
…
-
Hi,
I have replicated the training and evaluation for the pair_rm model, but I haven't achieved the results reported in Table 2 of the paper. The best results I obtained were with pm_models/llama3-…