Reproduce Table 7 in the Paper

[lunaryan]

Hi authors,

Thanks so much for your brilliant code!

I am trying to reproduce table 7. Using the default setting of test_all.sh, my results are: diff_data_dir:raphael-kirchner/clean/test/500/, ACC:94.85% diff_data_dir:camille-pissarro/clean/test/500/, ACC:100.00% diff_data_dir:pyotr-konchalovsky/clean/test/500/, ACC:96.94% diff_data_dir:childe-hassam/clean/test/500/, ACC:98.00% diff_data_dir:paul-cezanne/clean/test/500/, ACC:100.00% diff_data_dir:claude-monet/clean/test/500/, ACC:100.00% diff_data_dir:albrecht-durer/clean/test/500/, ACC:90.72% diff_data_dir:eugene-boudin/clean/test/500/, ACC:93.00%

diff_data_dir:raphael-kirchner/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:91.75% diff_data_dir:camille-pissarro/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:49.00% diff_data_dir:pyotr-konchalovsky/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:76.53% diff_data_dir:childe-hassam/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:96.00% diff_data_dir:paul-cezanne/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:79.59% diff_data_dir:claude-monet/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:97.00% diff_data_dir:albrecht-durer/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:86.60% diff_data_dir:eugene-boudin/adv_p0.05_alpha30_iter500_lr0.01/test/500_trans_Cubism_by_Picasso_seed0/, ACC:90.00%

where the acc of glaze is far from what are listed in Table 7. Is it possible to confirm that Table 7 uses the same hyper-parameters? Or what am I missing here? Thanks so much!

[AAAAAAsuka]

It seems that you are having trouble reproducing the Glaze experiment. Due to lack of information, I hope you can answer the following questions:

Did you directly check the result of the image generation, and does Glaze work? And does the loss converge normally when Glaze is performed?

If the answer is yes, have you tried the Diffusion Classifier? If yes, have you tried Diffusion Classifier?

Did you check the hyperparameter settings in detail to keep them consistent with the paper? One thing to note is that you need to keep the batch size to 32 when fine-tuning the SD model, and the actual batch size depends on the batch size per GPU, gradient accumulation steps and the number of GPUs. For example, in our experiment, we used 4 GPUs, with a per-GPU batch size of 8, and a gradient accumulation of 1, resulting in an actual batch size of 4 8 1 = 32. If you use 2 GPUs and a per-batch size of 8, you should change the gradient accumulation to 2 to maintain a consistent actual batch size of 32.

We spoke with the Glaze team before the code was released, and they checked our experimental code and did not dispute the reproduction results.

and here's my envirments:

Package                       Version              
----------------------------- -------------------- 
absl-py                       1.3.0
accelerate                    0.22.0
advertorch                    0.2.4
affine                        2.4.0
aiofiles                      23.2.1
aiohttp                       3.8.1
aiosignal                     1.2.0
alabaster                     0.7.12
altair                        5.0.1
anaconda-client               1.9.0
anaconda-navigator            2.1.4
anaconda-project              0.10.2
antlr4-python3-runtime        4.8
anyio                         3.5.0
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
astroid                       2.6.6
astropy                       5.0.4
asttokens                     2.0.5
astunparse                    1.6.3
async-timeout                 4.0.2
atomicwrites                  1.4.0
attrs                         22.1.0
autoattack                    0.1
Automat                       20.2.0
autopep8                      1.6.0
Babel                         2.11.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
backports.tempfile            1.0
backports.weakref             1.0.post1
bcrypt                        3.2.0
beautifulsoup4                4.11.1
binaryornot                   0.4.4
bitarray                      2.5.1
bkcharts                      0.2
black                         22.6.0
bleach                        4.1.0
bokeh                         2.4.2
boltons                       21.0.0
boto3                         1.21.32
botocore                      1.24.32
Bottleneck                    1.3.4
brotlipy                      0.7.0
cachetools                    4.2.2
category-encoders             2.2.2
certifi                       2022.12.7
cffi                          1.15.1
chardet                       4.0.0
charset-normalizer            2.0.4
click                         8.0.4
click-plugins                 1.1.1
cligj                         0.7.2
clip                          1.0              
cloudpickle                   2.0.0
clyent                        1.2.2
cmake                         3.25.2
colorama                      0.4.6
colorcet                      2.0.6
comm                          0.1.2
conda                         23.1.0
conda-build                   3.23.3
conda-content-trust           0.1.3
conda-pack                    0.6.0
conda-package-handling        2.0.2
conda_package_streaming       0.7.0
conda-repo-cli                1.0.4
conda-token                   0.3.0
conda-verify                  3.4.2
constantly                    15.1.0
contextlib2                   21.6.0
contourpy                     1.0.5
cookiecutter                  1.7.3
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
Cython                        0.29.33
cytoolz                       0.12.0
daal4py                       2021.5.0
dask                          2022.2.1
datasets                      2.10.0
datashader                    0.13.0
datashape                     0.5.4
debugpy                       1.5.1
decorator                     5.1.1
defusedxml                    0.7.1
diff-match-patch              20200713
diffusers                     0.17.0.dev0
dill                          0.3.6
distributed                   2022.2.1
dm-haiku                      0.0.10.dev0
docker-pycreds                0.4.0
docutils                      0.17.1
einops                        0.6.1
entrypoints                   0.4
et-xmlfile                    1.1.0
executing                     0.8.3
fairscale                     0.4.13
faiss-gpu                     1.7.2
fastapi                       0.101.1
fastjsonschema                2.16.2
ffmpy                         0.3.1
filelock                      3.9.0
fire                          0.5.0
flake8                        3.9.2
Flask                         2.2.2
flatbuffers                   2.0.7
flit_core                     3.6.0
fonttools                     4.25.0
frozenlist                    1.3.3
fschat                        0.2.20
fsspec                        2022.11.0
ftfy                          6.1.1
future                        0.18.2
fuzzywuzzy                    0.18.0
gast                          0.4.0
gdown                         4.6.4
gensim                        4.1.2
geotorch                      0.3.0
gitdb                         4.0.10
GitPython                     3.1.32
glob2                         0.7
gmpy2                         2.1.2
google-api-core               1.25.1
google-auth                   1.33.0
google-auth-oauthlib          0.4.6
google-cloud-core             1.7.1
google-cloud-storage          1.31.0
google-crc32c                 1.5.0
google-pasta                  0.2.0
google-resumable-media        1.3.1
googleapis-common-protos      1.56.4
gradio                        3.41.0
gradio_client                 0.5.0
greenlet                      2.0.1
grpcio                        1.42.0
h11                           0.14.0
h5py                          3.6.0
HeapDict                      1.0.1
holoviews                     1.14.8
httpcore                      0.17.3
httpx                         0.24.1
huggingface-hub               0.16.4
hvplot                        0.7.3
hydra-core                    1.0.7
hyperlink                     21.0.0
idna                          3.4
image-similarity-measures     0.3.5
imagecodecs                   2021.8.26
imageio                       2.9.0
imagesize                     1.4.1
importlib-metadata            4.11.3
importlib-resources           6.0.1
incremental                   21.3.0
inflection                    0.5.1
iniconfig                     1.1.1
intake                        0.6.5
intervaltree                  3.1.0
ipdb                          0.13.9
ipykernel                     6.19.2
ipython                       8.10.0
ipython-genutils              0.2.0
ipywidgets                    7.6.5
isort                         5.9.3
itemadapter                   0.3.0
itemloaders                   1.0.4
itsdangerous                  2.0.1
jax                           0.3.0
jaxlib                        0.3.0+cuda11.cudnn82
jdcal                         1.4.1
jedi                          0.18.1
jeepney                       0.7.1
Jinja2                        3.1.2
jinja2-time                   0.2.0
jmespath                      0.10.0
jmp                           0.0.2
joblib                        1.1.1
json5                         0.9.6
jsonlines                     3.1.0
jsonschema                    4.17.3
jupyter                       1.0.0
jupyter-client                6.1.12
jupyter-console               6.4.0
jupyter_core                  5.2.0
jupyter-server                1.13.5
jupyterlab                    3.3.2
jupyterlab-pygments           0.1.2
jupyterlab-server             2.10.3
jupyterlab-widgets            1.0.0
keras                         2.11.0
keyring                       23.4.0
kiwisolver                    1.4.4
kmeans-pytorch                0.3
kornia                        0.6.8
lazy-object-proxy             1.6.0
libarchive-c                  2.9
libclang                      14.0.6
lightly                       1.2.27
lightly-utils                 0.0.2
lit                           16.0.6
littleutils                   0.2.2
llm-attacks                   0.0.1               
llvmlite                      0.38.0
lmdb                          1.4.0
locket                        1.0.0
loguru                        0.5.3
loralib                       0.1.1
lpips                         0.1.4
lxml                          4.9.1
Markdown                      3.4.1
markdown-it-py                3.0.0
markdown2                     2.4.10
MarkupSafe                    2.1.1
matplotlib                    3.7.2
matplotlib-inline             0.1.6
mccabe                        0.6.1
mdurl                         0.1.2
mistune                       0.8.4
mkl-fft                       1.3.1
mkl-random                    1.2.2
mkl-service                   2.4.0
ml-collections                0.1.1
mock                          4.0.3
mpmath                        1.2.1
msgpack                       1.0.3
multidict                     6.0.2
multipledispatch              0.6.0
multiprocess                  0.70.14
munkres                       1.1.4
mypy-extensions               0.4.3
natsort                       8.2.0
navigator-updater             0.2.1
nbclassic                     0.3.5
nbclient                      0.5.13
nbconvert                     6.4.4
nbformat                      5.7.0
nest-asyncio                  1.5.6
networkx                      2.8.4
nh3                           0.2.14
ninja                         1.11.1
nltk                          3.7
nose                          1.3.7
notebook                      6.4.8
numba                         0.55.1
numexpr                       2.8.1
numpy                         1.24.4
numpydoc                      1.2
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
nvidia-cufft-cu11             10.9.0.58
nvidia-curand-cu11            10.2.10.91
nvidia-cusolver-cu11          11.4.0.1
nvidia-cusparse-cu11          11.7.4.91
nvidia-nccl-cu11              2.14.3
nvidia-nvtx-cu11              11.7.91
oauthlib                      3.2.2
objax                         1.6.0
ogb                           1.3.4
olefile                       0.46
omegaconf                     2.0.6
openai                        0.27.9
opencv-python                 4.6.0.66
openpyxl                      3.0.10
opt-einsum                    3.3.0
orjson                        3.9.5
outdated                      0.2.1
packaging                     22.0
pandas                        2.0.3
pandocfilters                 1.5.0
panel                         0.13.0
param                         1.12.3
parameterized                 0.8.1
parsel                        1.6.0
parso                         0.8.3
partd                         1.2.0
pathspec                      0.10.3
pathtools                     0.1.2
patsy                         0.5.2
peft                          0.5.0
pep8                          1.7.1
pexpect                       4.8.0
phasepack                     1.5
pickleshare                   0.7.5
Pillow                        9.3.0
pip                           22.3.1
pkginfo                       1.8.3
platformdirs                  2.5.2
plotly                        5.9.0
pluggy                        1.0.0
poyo                          0.5.0
prometheus-client             0.14.1
prompt-toolkit                3.0.36
Protego                       0.1.16
protobuf                      3.20.3
psutil                        5.9.0
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pyarrow                       11.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.7.0
pycosat                       0.6.4
pycparser                     2.21
pyct                          0.4.6
pycurl                        7.45.1
pydantic                      1.10.12
pyDeprecate                   0.3.0
PyDispatcher                  2.0.5
pydocstyle                    6.3.0
pydub                         0.25.1
pyerfa                        2.0.0
pyFFTW                        0.13.1
pyflakes                      2.3.1
Pygments                      2.16.1
PyHamcrest                    2.0.2
PyHessian                     0.1
PyJWT                         2.4.0
pylint                        2.9.6
pyls-spyder                   0.4.0
pyodbc                        4.0.34
pyOpenSSL                     22.0.0
pyparsing                     3.0.9
pyperclip                     1.8.2
pyre-extensions               0.0.23
pyrsistent                    0.18.0
PySocks                       1.7.1
PyTDC                         0.3.7
pytest                        7.1.2
python-dateutil               2.8.2
python-lsp-black              1.0.0
python-lsp-jsonrpc            1.0.0
python-lsp-server             1.2.4
python-multipart              0.0.6
python-slugify                5.0.2
python-snappy                 0.6.1
pytoolconfig                  1.2.5
pytorch-fid                   0.3.0
pytorch-ignite                0.4.12
pytorch-lightning             1.3.6
pytorch-tabnet                3.0.0
pytorch-tabular               0.7.0
pytorch-wavelet               1.2
pytorch-wavelets              1.3.0
pytz                          2022.7
pyviz-comms                   2.0.2
PyWavelets                    1.3.0
pyxdg                         0.27
PyYAML                        6.0
pyzmq                         23.2.0
QDarkStyle                    3.0.2
qstylizer                     0.2.2
QtAwesome                     1.2.2
qtconsole                     5.4.0
QtPy                          2.2.0
queuelib                      1.5.0
rank-bm25                     0.2.2
rasterio                      1.3.6
rdkit-pypi                    2022.3.5
regex                         2022.7.9
requests                      2.30.0
requests-file                 1.5.1
requests-oauthlib             1.3.1
responses                     0.18.0
rich                          13.5.2
robustbench                   1.1
rope                          1.7.0
rsa                           4.7.2
Rtree                         1.0.1
ruamel.yaml                   0.17.21
ruamel.yaml.clib              0.2.6
ruamel-yaml-conda             0.17.21
s3transfer                    0.5.0
safetensors                   0.3.3
scikit-image                  0.19.2
scikit-learn                  1.2.1
scikit-learn-intelex          2021.20220215.212715
scipy                         1.10.1
Scrapy                        2.6.1
seaborn                       0.11.2
SecretStorage                 3.3.1
semantic-version              2.10.0
Send2Trash                    1.8.0
sentence-transformers         2.2.2
sentencepiece                 0.1.97
sentry-sdk                    1.29.2
service-identity              18.1.0
setproctitle                  1.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
sewar                         0.4.5
shortuuid                     1.0.11
sip                           4.19.13
six                           1.16.0
smart-open                    5.1.0
smmap                         5.0.0
sniffio                       1.2.0
snowballstemmer               2.2.0
snuggs                        1.4.7
sortedcollections             2.1.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
Sphinx                        4.4.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
spyder                        5.1.5
spyder-kernels                2.1.3
SQLAlchemy                    1.4.39
stack-data                    0.2.0
starlette                     0.27.0
statsmodels                   0.13.2
svgwrite                      1.4.3
sympy                         1.11.1
tables                        3.6.1
tabulate                      0.8.10
taming-transformers           0.0.1               
TBB                           0.2
tblib                         1.7.0
tenacity                      8.0.1
tensorboard                   2.11.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorflow                    2.11.0
tensorflow-estimator          2.11.0
tensorflow-io-gcs-filesystem  0.29.0
termcolor                     2.2.0
terminado                     0.17.1
testpath                      0.6.0
text-unidecode                1.3
textdistance                  4.2.1
threadpoolctl                 2.2.0
three-merge                   0.1.1
tifffile                      2021.7.2
tiktoken                      0.4.0
timm                          0.6.12
tinycss                       0.4
tinycss2                      1.2.1
tldextract                    3.2.0
tokenizers                    0.13.3
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.0
torch                         2.0.1
torchaudio                    0.13.1
torchcontrib                  0.0.2
torchdiffeq                   0.2.3
torchmetrics                  0.6.0
torchsde                      0.2.5
torchvision                   0.14.1
tornado                       6.2
tqdm                          4.64.1
traitlets                     5.7.1
trampoline                    0.1.2
transformers                  4.28.1
triton                        2.0.0
Twisted                       22.2.0
typed-ast                     1.4.3
typing_extensions             4.7.1
typing-inspect                0.8.0
tzdata                        2023.3
ujson                         5.4.0
Unidecode                     1.2.0
urllib3                       1.26.16
uvicorn                       0.23.2
venn                          0.1.3
w3lib                         1.21.0
wandb                         0.15.8
warmup-scheduler              0.3
watchdog                      2.1.6
wavedrom                      2.0.3.post3
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.58.0
websockets                    11.0.3
Werkzeug                      2.2.2
wheel                         0.38.4
widgetsnbextension            3.5.2
wilds                         2.0.0
wrapt                         1.12.1
wurlitzer                     3.0.2
xarray                        0.20.1
xformers                      0.0.16
xlrd                          2.0.1
XlsxWriter                    3.0.3
xxhash                        3.2.0
yapf                          0.31.0
yarl                          1.6.3
zict                          2.1.0
zipp                          3.11.0
zope.interface                5.4.0
zstandard                     0.19.0

thanks!

[lunaryan]

Thanks for your reply.

I do manually check the results of glaze. I noticed the perturbation but it is hard to say whether it changes the style since I am not an artist and also impossible to tell how much the perturbation can change the diffusion classifier's predictions.

Yes, I tried the diffusion classifier and in my initial post I am showing exactly the results of diffusion classifier.

The decrease process of loss seems normal to me.

From appendix G I can only confirm I am using the same target style and it does not mention other hyper-parameters. I am using the default setting from test_all.sh and did not change anything (so it is 4 gpu with batch size 8 for each). So could you please kindly check whether the parameters in your test_all.sh are consistent with the setting in Table 7?

Thanks!

[AAAAAAsuka]

Hi.

All hyperparameters are located in Appendix B, not Appendix G. The experimental results in Appendix G are the result of performing Impress on a clean image, so the differences should be small by themselves.

I just checked test_all.sh, and I think the hyperparameter settings are the same as in the paper. I should note, however, that the number of graphics cards is not set in the test_all.sh file, but is specified by running accelerate config.

And I think there is a potential misunderstanding here, namely that the results of our experiments are not the classification of the original image and the image with Glaze added, but the classification of the image generated by the SD model fine-tuned using the original image, and the classification of the image generated by the SD model fine-tuned using the Glaze-protected image.

In other words, we will fine-tune the SD model using the original image and the Glaze-protected image separately, and then use these fine-tuned models to generate new images, and then test the style categorization results on these newly generated images. Glaze perturbations may not by themselves change the predictions of the diffusion classifier.

In general, images generated by SD models fine-tuned with Glaze-protected images have very distinctly different styles (e.g., as shown in Figs. 1, 4, and 10 of the paper), but have you found similarly distinct differences?

I hope this helps.

[lunaryan]

Oh I am so sorry I meant Appendix H.

Thanks for your clarification. I will reset accelerate config and re-run the experiments.

I understand that the diffusion classifiers are applied to the generated images, as also indicated by the script. I think the term "perturbation" I used to describe the differences between generated images and clean test images causes the misunderstanding, my bad.

Yes, I can manually see that the generated images are different from the clean test images, and seem to have a style combining victim artist and target style. It is just hard to tell which style is the dominant -- that's why we need classifiers.

Anyway, thanks so much for your reply. I will rerun the experiments and lets see whether the classifiers give similar results to table 7 this time.

[AAAAAAsuka]

Thanks for the explanation, because I noticed that you used /test/500_trans_Cubism_by_Picasso_seed0/ when presenting the path to your data, which is the path I used early in writing the code, and which shouldn't exist when running the current version of test_all.sh. So I was guessing that an incorrect data source might be causing the problem.

The actual batch_size and accelerate settings are really complicated, and I usually monitor GPU usage while finetuning the SD model to make sure I'm using the right set of parameters. You can also check this in the log shown at the beginning of the finetuning SD code run.

If you re-run the experiment and it still doesn't work well, you can try increasing the Glaze iteration slightly, or tweaking other relevant parameters to get stronger protection. Also, you can use more finetuning steps when using the image with Glaze finetuning model . In our experiments, 500 steps gets the best protection, but we are not sure if this number is affected by the hardware device or software version. (This is common in our LLM research. )

Feel free to contact me via email if you need to, and we can try to set up a Zoom meeting to troubleshoot potential problems.

[lunaryan]

Oh yeah, about the directory name, I remembered that in the script it is something like transNum${transNum}_seed${SEED}. When I ran the script it throws an error complaining about no such directory. Then I checked the style_transfer.py and it seems the transferred images are under " save_dir = re.sub('clean', 'trans', copy.deepcopy(aimdir)) + f"/trans{re.sub(' ', '_', copy.deepcopy(aim_style_prompt))}_seed{args.manual_seed}/". I thought maybe the script was not up-to-date or something like that, so I just changed all the directory names in the scripts and there is no error anymore. Did I mess up anything?

Thanks for your suggestions about accelerate. I just set the batch size to 32 and use 1 gpu for simplicity this time.

I will try your advice about glaze iterations, other protection-strength related parameters, as well as the finetuning steps if it still does not work. Those are helpful information and I really appreciate it.

Thanks so much for your support! You are so nice!

[AAAAAAsuka]

Thanks for the feedback, I don't think this will lead to any fatal errors (especially if you've already checked the experimental results), but I'll check over test_all.sh at a later date to correct the potential path error.

Since I've already been informed of some successful reproductions, I'd also be interested in learning about any possible causes of unsuccessful reproduction of our results, and correcting them in time to prevent more people from experiencing these problems :)

You're so welcome, please feel free to contact me if you have any other questions.