search

hongcheki / sweet-watermark

Official repository of the paper: Who Wrote this Code? Watermarking for Code Generation (ACL 2024)
https://arxiv.org/abs/2305.15060
29 stars 4 forks source link

DS-1000 Accuracy is 0 #10

Open TimeLovercc opened 1 week ago

TimeLovercc commented 1 week ago

I've used the scripts in scripts/main/. My model is starcode2-3B. I just find that the accuracy is 0. Is there any error in the code?

thynl commented 2 days ago

Hi, our codebase is SWEET watermark on bigcode-evaluation-harness repository. Could you use that repository's environment when detecting the watermark & evaluating the code performance (generation doesn't matter) and tell me if it solves the problem?

pip install -e ".[ds1000]" # installs all additional dependencies except PyTorch
# torch==1.12.1 required. Download version with relevant GPU support etc., e.g.,
pip install torch==1.12.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# to suppress any tensorflow optimization warnings, 
# precede call to "accelerate launch" with "TF_CPP_MIN_LOG_LEVEL=3"

# on some systems, tensorflow will attempt to allocate all GPU memory
# to its process at import which will raise a CUDA out-of-memory error
# setting "export TF_FORCE_GPU_ALLOW_GROWTH=true" resolves this