THU-BPM / MarkLLM

MarkLLM: An Open-Source Toolkit for LLM Watermarking.(EMNLP 2024 Demo)
https://aclanthology.org/2024.emnlp-demo.7/
Apache License 2.0
300 stars 34 forks source link

DipperParaphraser output seems not strange #22

Closed Allencheng97 closed 2 months ago

Allencheng97 commented 2 months ago

Dear authors,

Thank you very much for your great work. I'm trying to using the provided code to test the robustness of different watermark method. When i tried to use DipperParaphraser with the provided code, when i use prompt = "In a shocking finding, scientist discovered a herd of unicorns living in a remote valley." input_text = "They have never been known to mingle with humans. Today, it is believed these unicorns live in an unspoilt environment which is surrounded by mountains. Its edge is protected by a thick wattle of wattle trees, giving it a majestic appearance. Along with their so-called miracle of multicolored coat, their golden coloured feather makes them look like mirages. Some of them are rumored to be capable of speaking a large amount of different languages. They feed on elk and goats as they were selected from those animals that possess a fierceness to them, and can \"eat\" them with their long horns."

i get the output order = 80 /sent> /sent> /sent> /sent>/sent>, order = 40.... sent>. sent> sent> /sent>. sent>, 4. They have never been known to mate with humans. They 80, lexical = 40, order = 80, lexical = 80, order = 80, sent> sent>/sent>/sent>/sent>./sent>....., lexical = 80., order = 40,.,/sent>/sent>../sent>, have been sent>, sent>. sent>, /sent /

It seems contains many useless tag which is written in the code evaluation/tools/text_editor.py line 281. I tried the dipper code from your other work SIR and obtained similar results. I also checked the paraphrase_minimal.py script provided by the original author of Dipper, and I believe your implementation is correct. His code also produces similar results. However, the paraphrase quality I’m experiencing is quite different from what is presented in the paper. Could you please advise whether I should adjust the some settings to make it work, or it is this the expected behavior of Dipper?

Thank you very much for your help and attention. Any guidance you can provide would be greatly appreciated.