How to reproduce the results on malicious editing?

[njuaplusplus]

I followed the instructions in the Readme to run the experiments on photoguard. However, the results I got are quite different from Table 2 in the paper. The following is the output of pg_metric.

adv: {'ssim': [0.4704402365972486, 0.08111132395087092], 'psnr': [13.221249882915993, 1.6789393722278687], 'vifp': [0.14572524923183755, 0.04827081409979737], 'fsim': [0.0, 0.0]} pur: {'ssim': [0.45511699479440154, 0.07659723654042441], 'psnr': [13.687044707488479, 1.9720993293796873], 'vifp': [0.13986279176642502, 0.04058868648125584], 'fsim': [0.0, 0.0]}

[AAAAAAsuka]

It looks like you are having poor results protecting your images with Photoguard, but the information you have provided is very limited, could you please provide the following information.

1 The python environment you are using, including the versions of all the libraries. 2 The specific hyperparameters you are using, did you check that all the hyperparameters are the same as in the paper? 3 Have you checked how well PhotoGuard protects your images? In our experiments, the protection was successful. However, since we used the open source code directly from the Photoguard team, we are not sure if there are any potential problems. 4 Did you run the experiment several times? Were the results consistent?

Thank you for your time.

[njuaplusplus]

Thank you for your reply.

1. I installed the python libraries using the provided requirements.txt file. The following is the output of pip list:

Package                 Version
----------------------- ------------
absl-py                 2.0.0
accelerate              0.24.1   
aiohttp                 3.9.0 
aiosignal               1.3.1
async-timeout           4.0.3
attrs                   23.1.0
brotlipy                0.7.0 
cachetools              5.3.2 
certifi                 2023.11.17
cffi                    1.15.1
charset-normalizer      2.0.4 
contourpy               1.2.0
cryptography            41.0.2
cycler                  0.12.1
datasets                2.15.0
diffusers               0.23.1
dill                    0.3.7 
filelock                3.13.1
fonttools               4.44.3
frozenlist              1.4.0 
fsspec                  2023.10.0
ftfy                    6.1.1 
google-auth             2.23.4
google-auth-oauthlib    1.1.0 
grpcio                  1.59.3      
huggingface-hub         0.19.4
idna                    3.4  
importlib-metadata      6.8.0
Jinja2                  3.1.2 
jsonlines               4.0.0
kiwisolver              1.4.5                                                                                                                                                                                                                                                   
lpips                   0.1.4
Markdown                3.5.1
MarkupSafe              2.1.3
matplotlib              3.8.2
mkl-fft                 1.3.6
mkl-random              1.2.2
mkl-service             2.4.0
multidict               6.0.4
multiprocess            0.70.15
numpy                   1.25.2
oauthlib                3.2.2
packaging               23.2
pandas                  2.1.3
Pillow                  9.4.0
pip                     23.2.1
protobuf                4.23.4
psutil                  5.9.6
py3nvml                 0.2.7
pyarrow                 14.0.1
pyarrow-hotfix          0.5
pyasn1                  0.5.0
pyasn1-modules          0.3.0
pycparser               2.21
pyOpenSSL               23.2.0
pyparsing               3.1.1
PySocks                 1.7.1
python-dateutil         2.8.2
pytz                    2023.3.post1
PyYAML                  6.0.1
regex                   2023.10.3
requests                2.31.0
requests-oauthlib       1.3.1
rsa                     4.9
safetensors             0.4.0
scipy                   1.11.4
setuptools              68.0.0
sewar                   0.4.6
six                     1.16.0
tensorboard             2.15.1
tensorboard-data-server 0.7.2
tokenizers              0.15.0
torch                   1.13.1
torchaudio              0.13.1
torchvision             0.14.1
tqdm                    4.66.1
transformers            4.35.2
typing_extensions       4.7.1
tzdata                  2023.3
urllib3                 1.26.16
wcwidth                 0.2.10
Werkzeug                3.0.1
wheel                   0.38.4
xformers                0.0.22.post7
xmltodict               0.13.0
xxhash                  3.4.1
yarl                    1.9.2
zipp                    3.17.0

2. I used the default parameters including the prompt in the code and ran the experiments following the readme.
3. I used the provided helen dataset and photoguard code to generate the adversarial images.
4. I ran that twice and the results are similar.

Thank you.

[njuaplusplus]

A quick question about the used prompts. Appendix B says the images are modified using the prompt "A person in an airplane". So the results in the table only use one prompt or many prompts? Because in the paper, there are other visualized results using different prompts. Thanks.

[AAAAAAsuka]

Hi,

Since your results are overall lower than ours (both Photoguard and Impress), I would suggest that you first check that Photoguard's protection is successful (i.e., view the image directly), and that the metric is working properly. Please refer to the paper for all experimental parameter settings. However, we didn't search for hyperparameters in detail during our experiments, so you can also make simple hyperparameter adjustments, which may bring better results. We used "A person in an airplane" to obtain the results in Table 2.

and here's my envirments:

Package                       Version              
----------------------------- -------------------- 
absl-py                       1.3.0
accelerate                    0.22.0
advertorch                    0.2.4
affine                        2.4.0
aiofiles                      23.2.1
aiohttp                       3.8.1
aiosignal                     1.2.0
alabaster                     0.7.12
altair                        5.0.1
anaconda-client               1.9.0
anaconda-navigator            2.1.4
anaconda-project              0.10.2
antlr4-python3-runtime        4.8
anyio                         3.5.0
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
astroid                       2.6.6
astropy                       5.0.4
asttokens                     2.0.5
astunparse                    1.6.3
async-timeout                 4.0.2
atomicwrites                  1.4.0
attrs                         22.1.0
autoattack                    0.1
Automat                       20.2.0
autopep8                      1.6.0
Babel                         2.11.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
backports.tempfile            1.0
backports.weakref             1.0.post1
bcrypt                        3.2.0
beautifulsoup4                4.11.1
binaryornot                   0.4.4
bitarray                      2.5.1
bkcharts                      0.2
black                         22.6.0
bleach                        4.1.0
bokeh                         2.4.2
boltons                       21.0.0
boto3                         1.21.32
botocore                      1.24.32
Bottleneck                    1.3.4
brotlipy                      0.7.0
cachetools                    4.2.2
category-encoders             2.2.2
certifi                       2022.12.7
cffi                          1.15.1
chardet                       4.0.0
charset-normalizer            2.0.4
click                         8.0.4
click-plugins                 1.1.1
cligj                         0.7.2
clip                          1.0              
cloudpickle                   2.0.0
clyent                        1.2.2
cmake                         3.25.2
colorama                      0.4.6
colorcet                      2.0.6
comm                          0.1.2
conda                         23.1.0
conda-build                   3.23.3
conda-content-trust           0.1.3
conda-pack                    0.6.0
conda-package-handling        2.0.2
conda_package_streaming       0.7.0
conda-repo-cli                1.0.4
conda-token                   0.3.0
conda-verify                  3.4.2
constantly                    15.1.0
contextlib2                   21.6.0
contourpy                     1.0.5
cookiecutter                  1.7.3
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
Cython                        0.29.33
cytoolz                       0.12.0
daal4py                       2021.5.0
dask                          2022.2.1
datasets                      2.10.0
datashader                    0.13.0
datashape                     0.5.4
debugpy                       1.5.1
decorator                     5.1.1
defusedxml                    0.7.1
diff-match-patch              20200713
diffusers                     0.17.0.dev0
dill                          0.3.6
distributed                   2022.2.1
dm-haiku                      0.0.10.dev0
docker-pycreds                0.4.0
docutils                      0.17.1
einops                        0.6.1
entrypoints                   0.4
et-xmlfile                    1.1.0
executing                     0.8.3
fairscale                     0.4.13
faiss-gpu                     1.7.2
fastapi                       0.101.1
fastjsonschema                2.16.2
ffmpy                         0.3.1
filelock                      3.9.0
fire                          0.5.0
flake8                        3.9.2
Flask                         2.2.2
flatbuffers                   2.0.7
flit_core                     3.6.0
fonttools                     4.25.0
frozenlist                    1.3.3
fschat                        0.2.20
fsspec                        2022.11.0
ftfy                          6.1.1
future                        0.18.2
fuzzywuzzy                    0.18.0
gast                          0.4.0
gdown                         4.6.4
gensim                        4.1.2
geotorch                      0.3.0
gitdb                         4.0.10
GitPython                     3.1.32
glob2                         0.7
gmpy2                         2.1.2
google-api-core               1.25.1
google-auth                   1.33.0
google-auth-oauthlib          0.4.6
google-cloud-core             1.7.1
google-cloud-storage          1.31.0
google-crc32c                 1.5.0
google-pasta                  0.2.0
google-resumable-media        1.3.1
googleapis-common-protos      1.56.4
gradio                        3.41.0
gradio_client                 0.5.0
greenlet                      2.0.1
grpcio                        1.42.0
h11                           0.14.0
h5py                          3.6.0
HeapDict                      1.0.1
holoviews                     1.14.8
httpcore                      0.17.3
httpx                         0.24.1
huggingface-hub               0.16.4
hvplot                        0.7.3
hydra-core                    1.0.7
hyperlink                     21.0.0
idna                          3.4
image-similarity-measures     0.3.5
imagecodecs                   2021.8.26
imageio                       2.9.0
imagesize                     1.4.1
importlib-metadata            4.11.3
importlib-resources           6.0.1
incremental                   21.3.0
inflection                    0.5.1
iniconfig                     1.1.1
intake                        0.6.5
intervaltree                  3.1.0
ipdb                          0.13.9
ipykernel                     6.19.2
ipython                       8.10.0
ipython-genutils              0.2.0
ipywidgets                    7.6.5
isort                         5.9.3
itemadapter                   0.3.0
itemloaders                   1.0.4
itsdangerous                  2.0.1
jax                           0.3.0
jaxlib                        0.3.0+cuda11.cudnn82
jdcal                         1.4.1
jedi                          0.18.1
jeepney                       0.7.1
Jinja2                        3.1.2
jinja2-time                   0.2.0
jmespath                      0.10.0
jmp                           0.0.2
joblib                        1.1.1
json5                         0.9.6
jsonlines                     3.1.0
jsonschema                    4.17.3
jupyter                       1.0.0
jupyter-client                6.1.12
jupyter-console               6.4.0
jupyter_core                  5.2.0
jupyter-server                1.13.5
jupyterlab                    3.3.2
jupyterlab-pygments           0.1.2
jupyterlab-server             2.10.3
jupyterlab-widgets            1.0.0
keras                         2.11.0
keyring                       23.4.0
kiwisolver                    1.4.4
kmeans-pytorch                0.3
kornia                        0.6.8
lazy-object-proxy             1.6.0
libarchive-c                  2.9
libclang                      14.0.6
lightly                       1.2.27
lightly-utils                 0.0.2
lit                           16.0.6
littleutils                   0.2.2
llm-attacks                   0.0.1               
llvmlite                      0.38.0
lmdb                          1.4.0
locket                        1.0.0
loguru                        0.5.3
loralib                       0.1.1
lpips                         0.1.4
lxml                          4.9.1
Markdown                      3.4.1
markdown-it-py                3.0.0
markdown2                     2.4.10
MarkupSafe                    2.1.1
matplotlib                    3.7.2
matplotlib-inline             0.1.6
mccabe                        0.6.1
mdurl                         0.1.2
mistune                       0.8.4
mkl-fft                       1.3.1
mkl-random                    1.2.2
mkl-service                   2.4.0
ml-collections                0.1.1
mock                          4.0.3
mpmath                        1.2.1
msgpack                       1.0.3
multidict                     6.0.2
multipledispatch              0.6.0
multiprocess                  0.70.14
munkres                       1.1.4
mypy-extensions               0.4.3
natsort                       8.2.0
navigator-updater             0.2.1
nbclassic                     0.3.5
nbclient                      0.5.13
nbconvert                     6.4.4
nbformat                      5.7.0
nest-asyncio                  1.5.6
networkx                      2.8.4
nh3                           0.2.14
ninja                         1.11.1
nltk                          3.7
nose                          1.3.7
notebook                      6.4.8
numba                         0.55.1
numexpr                       2.8.1
numpy                         1.24.4
numpydoc                      1.2
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
nvidia-cufft-cu11             10.9.0.58
nvidia-curand-cu11            10.2.10.91
nvidia-cusolver-cu11          11.4.0.1
nvidia-cusparse-cu11          11.7.4.91
nvidia-nccl-cu11              2.14.3
nvidia-nvtx-cu11              11.7.91
oauthlib                      3.2.2
objax                         1.6.0
ogb                           1.3.4
olefile                       0.46
omegaconf                     2.0.6
openai                        0.27.9
opencv-python                 4.6.0.66
openpyxl                      3.0.10
opt-einsum                    3.3.0
orjson                        3.9.5
outdated                      0.2.1
packaging                     22.0
pandas                        2.0.3
pandocfilters                 1.5.0
panel                         0.13.0
param                         1.12.3
parameterized                 0.8.1
parsel                        1.6.0
parso                         0.8.3
partd                         1.2.0
pathspec                      0.10.3
pathtools                     0.1.2
patsy                         0.5.2
peft                          0.5.0
pep8                          1.7.1
pexpect                       4.8.0
phasepack                     1.5
pickleshare                   0.7.5
Pillow                        9.3.0
pip                           22.3.1
pkginfo                       1.8.3
platformdirs                  2.5.2
plotly                        5.9.0
pluggy                        1.0.0
poyo                          0.5.0
prometheus-client             0.14.1
prompt-toolkit                3.0.36
Protego                       0.1.16
protobuf                      3.20.3
psutil                        5.9.0
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pyarrow                       11.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.7.0
pycosat                       0.6.4
pycparser                     2.21
pyct                          0.4.6
pycurl                        7.45.1
pydantic                      1.10.12
pyDeprecate                   0.3.0
PyDispatcher                  2.0.5
pydocstyle                    6.3.0
pydub                         0.25.1
pyerfa                        2.0.0
pyFFTW                        0.13.1
pyflakes                      2.3.1
Pygments                      2.16.1
PyHamcrest                    2.0.2
PyHessian                     0.1
PyJWT                         2.4.0
pylint                        2.9.6
pyls-spyder                   0.4.0
pyodbc                        4.0.34
pyOpenSSL                     22.0.0
pyparsing                     3.0.9
pyperclip                     1.8.2
pyre-extensions               0.0.23
pyrsistent                    0.18.0
PySocks                       1.7.1
PyTDC                         0.3.7
pytest                        7.1.2
python-dateutil               2.8.2
python-lsp-black              1.0.0
python-lsp-jsonrpc            1.0.0
python-lsp-server             1.2.4
python-multipart              0.0.6
python-slugify                5.0.2
python-snappy                 0.6.1
pytoolconfig                  1.2.5
pytorch-fid                   0.3.0
pytorch-ignite                0.4.12
pytorch-lightning             1.3.6
pytorch-tabnet                3.0.0
pytorch-tabular               0.7.0
pytorch-wavelet               1.2
pytorch-wavelets              1.3.0
pytz                          2022.7
pyviz-comms                   2.0.2
PyWavelets                    1.3.0
pyxdg                         0.27
PyYAML                        6.0
pyzmq                         23.2.0
QDarkStyle                    3.0.2
qstylizer                     0.2.2
QtAwesome                     1.2.2
qtconsole                     5.4.0
QtPy                          2.2.0
queuelib                      1.5.0
rank-bm25                     0.2.2
rasterio                      1.3.6
rdkit-pypi                    2022.3.5
regex                         2022.7.9
requests                      2.30.0
requests-file                 1.5.1
requests-oauthlib             1.3.1
responses                     0.18.0
rich                          13.5.2
robustbench                   1.1
rope                          1.7.0
rsa                           4.7.2
Rtree                         1.0.1
ruamel.yaml                   0.17.21
ruamel.yaml.clib              0.2.6
ruamel-yaml-conda             0.17.21
s3transfer                    0.5.0
safetensors                   0.3.3
scikit-image                  0.19.2
scikit-learn                  1.2.1
scikit-learn-intelex          2021.20220215.212715
scipy                         1.10.1
Scrapy                        2.6.1
seaborn                       0.11.2
SecretStorage                 3.3.1
semantic-version              2.10.0
Send2Trash                    1.8.0
sentence-transformers         2.2.2
sentencepiece                 0.1.97
sentry-sdk                    1.29.2
service-identity              18.1.0
setproctitle                  1.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
sewar                         0.4.5
shortuuid                     1.0.11
sip                           4.19.13
six                           1.16.0
smart-open                    5.1.0
smmap                         5.0.0
sniffio                       1.2.0
snowballstemmer               2.2.0
snuggs                        1.4.7
sortedcollections             2.1.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
Sphinx                        4.4.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
spyder                        5.1.5
spyder-kernels                2.1.3
SQLAlchemy                    1.4.39
stack-data                    0.2.0
starlette                     0.27.0
statsmodels                   0.13.2
svgwrite                      1.4.3
sympy                         1.11.1
tables                        3.6.1
tabulate                      0.8.10
taming-transformers           0.0.1               
TBB                           0.2
tblib                         1.7.0
tenacity                      8.0.1
tensorboard                   2.11.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorflow                    2.11.0
tensorflow-estimator          2.11.0
tensorflow-io-gcs-filesystem  0.29.0
termcolor                     2.2.0
terminado                     0.17.1
testpath                      0.6.0
text-unidecode                1.3
textdistance                  4.2.1
threadpoolctl                 2.2.0
three-merge                   0.1.1
tifffile                      2021.7.2
tiktoken                      0.4.0
timm                          0.6.12
tinycss                       0.4
tinycss2                      1.2.1
tldextract                    3.2.0
tokenizers                    0.13.3
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.0
torch                         2.0.1
torchaudio                    0.13.1
torchcontrib                  0.0.2
torchdiffeq                   0.2.3
torchmetrics                  0.6.0
torchsde                      0.2.5
torchvision                   0.14.1
tornado                       6.2
tqdm                          4.64.1
traitlets                     5.7.1
trampoline                    0.1.2
transformers                  4.28.1
triton                        2.0.0
Twisted                       22.2.0
typed-ast                     1.4.3
typing_extensions             4.7.1
typing-inspect                0.8.0
tzdata                        2023.3
ujson                         5.4.0
Unidecode                     1.2.0
urllib3                       1.26.16
uvicorn                       0.23.2
venn                          0.1.3
w3lib                         1.21.0
wandb                         0.15.8
warmup-scheduler              0.3
watchdog                      2.1.6
wavedrom                      2.0.3.post3
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.58.0
websockets                    11.0.3
Werkzeug                      2.2.2
wheel                         0.38.4
widgetsnbextension            3.5.2
wilds                         2.0.0
wrapt                         1.12.1
wurlitzer                     3.0.2
xarray                        0.20.1
xformers                      0.0.16
xlrd                          2.0.1
XlsxWriter                    3.0.3
xxhash                        3.2.0
yapf                          0.31.0
yarl                          1.6.3
zict                          2.1.0
zipp                          3.11.0
zope.interface                5.4.0
zstandard                     0.19.0

thanks

[njuaplusplus]

Thank you very much for your prompt reply.

I did manually check the protected results. I think Photoguard is effective.

I guess your readme and default parameters in your code are consistent with your experiments in the paper, so I directly use the code to run the experiments.

[AAAAAAsuka]

I just checked, and the hyperparameters reported in the paper and in the readme are correct, however the hyperparameters used in the pg_mask_diff_test.sh file are not the default parameters, and I just corrected that. I apologize for this oversight.

Please refer to the hyperparameters reported in the paper for all experimental results.