search

aiverify-foundation / moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
https://aiverify-foundation.github.io/moonshot/
Apache License 2.0
141 stars 25 forks source link

[Potential Bug] pt_id value set to None #270

Closed SidPad03 closed 1 month ago

SidPad03 commented 1 month ago

Steps to Reproduce

Run the CLI version of moonshot. Then create a new session with the toxic_sentence_generator attack module, and real-toxicity-prompt-template1 as the prompt template. Use any seed prompt as the input.

Details

My OS

Ubuntu 20.04.06 LTS

Error Log

moonshot (test-runner-toxic) [PT: real-toxicity-prompt-template1, CS: ]> run_attack_module toxic_sentence_generator "What is "
2024-07-24 23:32:19,104 [INFO][runner.py::run_red_teaming(476)] [Runner] test-runner-toxic - Running red teaming session...
2024-07-24 23:32:19,106 [INFO][session.py::__init__(290)] [Session] Session already exists.
2024-07-24 23:32:19,137 [INFO][redteaming.py::generate(92)] [Red teaming] Starting automated red teaming...
2024-07-24 23:32:22.150418: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-07-24 23:32:22.246321: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-24 23:32:23.162084: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-07-24 23:32:25,470 [INFO][redteaming.py::run_automated_red_teaming(161)] [Red teaming] Starting to run attack module [Toxic Sentence Generator]
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
2024-07-24 23:32:30,204 [INFO][connector.py::get_prediction(301)] Predicting prompt 1 [openai-gpt4]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-07-24 23:32:38,460 [ERROR][session.py::run(453)] [Session] Failed to run runner processing module in Part 3 due to error: 1 validation error for RedTeamingPromptArguments
pt_id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type
[run_attack_module]: str(1 validation error for RedTeamingPromptArguments
pt_id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type)
imda-kelvinkok commented 1 month ago

Hello @SidPad03. Thanks for the feedback!

We have fixed this issue and the fix will be scheduled for release next week (for the PyPI package). If you are using Moonshot from the source code, you can pull from this branch https://github.com/aiverify-foundation/moonshot/tree/ms-352_fix_CLI_ART_pt_id and try again.

imda-benedictlee commented 1 month ago

Hi @SidPad03, can I check if the fix works for you?

SidPad03 commented 1 month ago

Hi @imda-benedictlee,

This issue still persists after my testing of moonshot today. I pulled from the https://github.com/aiverify-foundation/moonshot/tree/ms-352_fix_CLI_ART_pt_id branch.

2024-07-29 21:23:33,459 [INFO][connector.py::get_prediction(301)] Predicting prompt 1 [openai-gpt4]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-07-29 21:23:38,591 [ERROR][session.py::run(453)] [Session] Failed to run runner processing module in Part 3 due to error: 1 validation error for RedTeamingPromptArguments
pt_id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type
[run_attack_module]: str(1 validation error for RedTeamingPromptArguments
pt_id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type)
imda-kelvinkok commented 1 month ago

Hi @SidPad03,

Can I check if you're using the library version of Moonshot, or the source code? You can do a pip freeze and see if you have aiverify-moonshot in your your Python environment. If you have it in your environment, you're using the library version and the fix on the branch will not affect the library as we have no released the fix for the libary.

Can I also confirm that your moonshot-data is updated (on main branch and pulled)?

SidPad03 commented 1 month ago

Hi @imda-kelvinkok,

Below is the output of pip freeze. I do have the aiverify-moonshot in my environment. How do I go about running the source code instead of the library version? I just followed the instructions listed in the readme.md.

Yes, my moonshot-data is pulled from the main branch.

(moonshot) vagrant@vm19:~/code$ pip freeze
absl-py==2.1.0
accelerate==0.30.1
aiohttp==3.9.5
aiometer==0.5.0
aiosignal==1.3.1
aiverify-moonshot==0.4.2
annotated-types==0.7.0
anthropic==0.25.6
anyio==4.4.0
anytree==2.12.1
astunparse==1.6.3
attrs==23.2.0
beautifulsoup4==4.12.3
bert-score==0.3.13
blis==0.7.11
boto3==1.34.114
botocore==1.34.114
bpemb==0.3.5
catalogue==2.0.10
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
cloudpathlib==0.16.0
cloudpickle==3.0.0
cmd2==2.4.3
confection==0.1.4
conllu==4.5.3
contourpy==1.2.1
cycler==0.12.1
cymem==2.0.8
datasets==2.19.1
dependency-injector==4.41.0
Deprecated==1.2.14
dill==0.3.8
distro==1.9.0
docopt==0.6.2
editdistance==0.8.1
eval_type_backport==0.2.0
fastapi==0.110.3
filelock==3.14.0
flair==0.12.2
flatbuffers==24.3.25
fonttools==4.52.4
frozenlist==1.4.1
fsspec==2024.3.1
ftfy==6.2.0
future==1.0.0
gast==0.5.4
gdown==4.4.0
gensim==4.3.2
google-pasta==0.2.0
grpcio==1.64.0
h11==0.14.0
h5py==3.11.0
homoglyphs==2.0.4
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.23.2
hyperopt==0.2.7
idna==3.7
ijson==3.2.3
Janome==0.5.0
jieba==0.42.1
Jinja2==3.1.4
jmespath==1.0.1
joblib==1.4.2
keras==3.3.3
kiwisolver==1.4.5
langcodes==3.4.0
langdetect==1.0.9
language-tool-python==2.8
language_data==1.2.0
lemminflect==0.2.3
libclang==18.1.1
lru-dict==1.3.0
lxml==5.2.2
marisa-trie==1.1.1
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.0
mdurl==0.1.2
ml-dtypes==0.3.2
more-itertools==10.2.0
mpld3==0.3
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
murmurhash==1.0.10
namex==0.0.8
networkx==3.3
nltk==3.8.1
num2words==0.5.13
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvtx-cu12==12.1.105
openai==1.30.3
OpenHowNet==2.0
opt-einsum==3.3.0
optree==0.11.0
packaging==24.0
pandas==2.2.2
pillow==10.3.0
pinyin==0.4.0
pptree==3.1
preshed==3.0.9
protobuf==4.25.3
psutil==5.9.8
py-readability-metrics==1.4.5
py4j==0.10.9.7
pyarrow==16.1.0
pyarrow-hotfix==0.6
pybind11==2.12.0
pydantic==2.7.2
pydantic_core==2.18.3
Pygments==2.18.0
pylcs==0.1.1
pyparsing==3.1.2
pyperclip==1.9.0
PySocks==1.7.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-slugify==8.0.4
pytorch_revgrad==0.2.0
pytz==2024.1
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.2
rich==13.7.1
rouge==1.0.1
rouge_score==0.1.2
s3transfer==0.10.1
safetensors==0.4.3
scikit-learn==1.5.0
scipy==1.12.0
segtok==1.5.11
sentence-transformers==2.7.0
sentencepiece==0.2.0
six==1.16.0
smart-open==6.4.0
sniffio==1.3.1
soupsieve==2.5
spacy==3.7.4
spacy-legacy==3.0.12
spacy-loggers==1.0.5
sqlitedict==2.1.0
srsly==2.4.8
starlette==0.37.2
sympy==1.12
tabulate==0.9.0
tensorboard==2.16.2
tensorboard-data-server==0.7.2
tensorflow==2.16.1
tensorflow-hub==0.16.1
tensorflow-io-gcs-filesystem==0.37.0
termcolor==2.4.0
terminaltables==3.1.10
text-unidecode==1.3
textattack==0.3.10
tf_keras==2.16.0
thinc==8.2.3
threadpoolctl==3.5.0
together==1.2.0
tokenizers==0.19.1
torch==2.3.0
tqdm==4.66.4
transformer-smaller-training-vocab==0.4.0
transformers==4.41.1
triton==2.3.0
typer==0.9.4
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
uvicorn==0.29.0
wasabi==1.1.2
wcwidth==0.2.13
weasel==0.3.4
Werkzeug==3.0.3
Wikipedia-API==0.6.0
word2number==1.1
wrapt==1.16.0
xgboost==2.0.3
xxhash==3.4.1
yarl==1.9.4
imda-kelvinkok commented 1 month ago

Hi @SidPad03,

We should be updating the library in a week or two.

If you want to try out the fix now on the codes, can you try the following:

  1. Do a git clone of the Moonshot repo in perhaps a new folder: git clone -b ms-352_fix_CLI_ART_pt_id git@github.com:aiverify-foundation/moonshot.git
  2. Optional but I'd advise you to create your own virtual environment: python -m venv venv. Activate the virtual env: source venv/bin/activate.
  3. Cd to the root Moonshot folder (you should see a lot of files including requirements.txt). Do a pip install -r requirements.txt.
  4. In the same directory: enter python -m moonshot -i moonshot-data. You will have the moonshot-data installed and the setup should be the same as the one you have when you followed the README.md. You can then try the CLI using this copy and see if it works.
SidPad03 commented 1 month ago

Hi @imda-kelvinkok,

I appreciate your response. The fix works now!