Table of Contents

• Updates
• Central Idea
• Motivation
• Limitation of existing works
• Pipeline
• Instructions for Code usage
• Citation

CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search [CVPR 2023]

Fahad Shamshad, Muzammal Naseer, Karthik Nandakumar
MBZUAI, UAE.

## Updates :loudspeaker: - **July-19** : Code released. - **June-19** : Code and demo release coming soon. Stay tuned! ## 🎯 Central Idea 🎯 ### We all love sharing photos online, but do you know big companies and even governments can use sneaky 🕵️‍♂️ face recognition software to track us? Our research takes this challenge head-on with a simple and creative idea 🌟: using carefully crafted makeup 💄 to outsmart the tracking software. The cherry on top? We're using everyday, easy-to-understand language 🗣️ to guide the makeup application, giving users much more flexibility! Our approach keeps your photos safe 🛡️ from unwanted trackers without making you look weird or having bizarre patches on your face, issues commonly seen with previous solutions. ## Motivation :muscle: :fire: - Malicious black-box Face recognition systems pose a serious threat to personal security/privacy of **5 billions people** using social media. - Unauthorized entities can use FR systems to **track user activities** by scraping face images from social media platforms. - There is an urgent demand for effective privacy preservation methods.

## Limitation of existing works :warning: - Recent noise-based facial privacy protection approaches result in artefacts. - Patch-based privacy approaches provide low privacy protection and their large visible pattern compromises naturalness.

## Pipeline **CLIP2Protect** generates face images that look natural and real. **But here's the special part**: it also ensures a high level of privacy protection. This means you can keep sharing images without worrying about unwanted tracking. It consists of two stages. - **The latent code initialization stage** reconstructs the given face image in the latent space by fine-tuning the generative model. - **The text-guided adversarial optimization stage** utilizes user-defined makeup text prompts and identity-preserving regularization to guide the search for adversarial codes within the latent space to effectively protect the facial privacy.

## Intructions for Code usage ### Setup - **Get code** ```shell git clone https://github.com/fahadshamshad/Clip2Protect.git ``` - **Build environment** ```shell cd Clip2Protect # use anaconda to build environment conda create -n clip2protect python=3.8 conda activate clip2protect # install packages pip install -r requirements.txt ``` ## Steps for Protecting Faces 1. Our solution relies on the [Rosinality](https://github.com/rosinality/stylegan2-pytorch/) PyTorch implementation of StyleGAN2. 2. **Download the pre-trained StyleGAN2 weights**: - Download the pre-trained StyleGAN2 weights from [here](https://drive.google.com/file/d/1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT/view?usp=sharing). - Place the weights in the 'pretrained_models' folder. 3. **Download pretrained face recognition models and dataset instructions**: - To acquire pretrained face recognition models and dataset instructions, including target images, please refer to the AMT-GAN page [here](https://github.com/CGCL-codes/AMT-GAN). - Place the pretrained face recognition model in the `models` folder. 4. **Acquire latent codes**: - We assume the latent codes are available in the `latents.pt` file. - You can acquire the latent codes of the face images to be protected using the encoder4editing (e4e) method available [here](https://github.com/omertov/encoder4editing). 5. **Run the code**: - The core functionality is in `main.py`. - Provide the `latents.pt` file and the corresponding faces directory, named 'input_images'. - Generate the protected faces in the 'results' folder by running the following command: ```shell python main.py --data_dir input_images --latent_path latents.pt --protected_face_dir results ``` 6. **Generator finetuning and adversarial optimization stages**: - The generator finetuning is implemented in `pivot_tuning.py`. - The adversarial optimization is implemented in `adversarial_optimization.py`. ## Citation If you're using CLIP2Protect in your research or applications, please cite using this BibTeX: ```bibtex @inproceedings{shamshad2023clip2protect, title={CLIP2Protect: Protecting Facial Privacy Using Text-Guided Makeup via Adversarial Latent Search}, author={Shamshad, Fahad and Naseer, Muzammal and Nandakumar, Karthik}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={20595--20605}, year={2023} } ```